Reinforced Optimal Estimator⁎⁎This work is supported by International Science & Technology Cooperation Program of China under 2019YFE0100200, NSF China with 51575293, and U20A20334. It is also partially supported by Geekplus Technology Co., Ltd

• Published in: IFAC-PapersOnLine Vol. 54; no. 20; pp. 366 - 373
• Main Authors: Cao, Wenhan, Chen, Jianyu, Duan, Jingliang, Li, Shengbo Eben, Lyu, Yao, Gu, Ziqing, Zhang, Yuhang
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 2021
• Subjects: minimum variance estimation; recurrent neural network; reinforcement learning; State estimation; time-invariant filter; recurrent neural network; minimum variance estimation; time-invariant filter; State estimation; reinforcement learning
• ISSN: 2405-8963
• DOI: 10.1016/j.ifacol.2021.11.201

Estimating the state of a stochastic system is a long-lasting issue in the areas of engineering and science. Existing methods either use approximations or yield a high computation burden. In this paper, we propose reinforced optimal estimator (ROE), which is an offline estimator for general nonlinear and non-Gaussian stochastic models. This method solves optimal estimation problems offline, and the learned estimator can be applied online efficiently. Firstly, we demonstrate that minimum variance estimation requires us to solve the estimation problem online, which causes low computation efficiency To overcome this drawback, we propose an infinite horizon optimal estimation problem, called reinforcement estimation problem, to obtain the offline estimator. The time-invariant filter of linear systems is shown as an example to analyze the equivalence between reinforcement estimation problem and minimum variance estimation problem. We show that such equivalence can only be found for linear systems, and the proposed problem formulation actually enables us to find the time-invariant estimator for general nonlinear systems. Then, we propose the ROE algorithm, inspired by reinforcement learning, and develop an actor-critic architecture to find a nearly optimal estimator of the reinforcement estimation problem. The estimator is approximated by recurrent neural networks, which has high online computation efficiency. The convergence is proved using contraction mapping and extended policy improvement theorem. Experiment results on complex nonlinear system estimation problems show that our method achieves higher estimation accuracy and computation efficiency than the unscented Kalman filter and particle filter.