MEC-A Near-Optimal Online Reinforcement Learning Algorithm for Continuous Deterministic Systems

• Published in: IEEE transaction on neural networks and learning systems Vol. 26; no. 2; pp. 346 - 356
• Main Authors: Zhao, Dongbin, Zhu, Yuanheng
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.02.2015; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithm design and analysis; Algorithms; Approximation algorithms; Computer simulation; Dynamical systems; Dynamics; Efficient exploration; Exploration; Heuristic algorithms; Learning; Learning systems; Mathematical analysis; Neural networks; Partitioning algorithms; Policies; Polynomials; probably approximately correct (PAC); reinforcement learning (RL); state aggregation; Upper bound; Efficient exploration; probably approximately correct (PAC); state aggregation; reinforcement learning (RL)
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2014.2371046

In this paper, the first probably approximately correct (PAC) algorithm for continuous deterministic systems without relying on any system dynamics is proposed. It combines the state aggregation technique and the efficient exploration principle, and makes high utilization of online observed samples. We use a grid to partition the continuous state space into different cells to save samples. A near-upper Q operator is defined to produce a near-upper Q function using samples in each cell. The corresponding greedy policy effectively balances between exploration and exploitation. With the rigorous analysis, we prove that there is a polynomial time bound of executing nonoptimal actions in our algorithm. After finite steps, the final policy reaches near optimal in the framework of PAC. The implementation requires no knowledge of systems and has less computation complexity. Simulation studies confirm that it is a better performance than other similar PAC algorithms.