Learning Algorithms for Markov Decision Processes with Average Cost

• Published in: SIAM journal on control and optimization Vol. 40; no. 3; pp. 681 - 698
• Main Authors: Abounadi, J., Bertsekas, D., Borkar, V. S.
• Format: Journal Article
• Language: English
• Published: Philadelphia, PA Society for Industrial and Applied Mathematics 01.01.2001
• Subjects: Algorithms; Applied sciences; Approximation; Artificial intelligence; Computer science; control theory; systems; Dynamic programming; Exact sciences and technology; Learning and adaptive systems; Markov analysis; Mathematics; Probability and statistics; Probability theory and stochastic processes; Random variables; Sciences and techniques of general use; Simulation; Stochastic processes; Markov process; Markov decision; Probabilistic approach; Average cost; Shortest path; Stochastic approximation; Time scale; Numerical method; Cost control; Markov chain; Asymptotic stability; Asynchronism; Learning (artificial intelligence); Asymptotic approximation; Learning algorithm; Artificial intelligence
• ISSN: 0363-0129; 1095-7138
• DOI: 10.1137/S0363012999361974

This paper gives the first rigorous convergence analysis of analogues of Watkins's Q-learning algorithm, applied to average cost control of finite-state Markov chains. We discuss two algorithms which may be viewed as stochastic approximation counterparts of two existing algorithms for recursively computing the value function of the average cost problem---the traditional relative value iteration (RVI) algorithm and a recent algorithm of Bertsekas based on the stochastic shortest path (SSP) formulation of the problem. Both synchronous and asynchronous implementations are considered and analyzed using the ODE method. This involves establishing asymptotic stability of associated ODE limits. The SSP algorithm also uses ideas fromtwo-time-scale stochastic approximation.