Distributed Neural Learning Algorithms for Multiagent Reinforcement Learning

• Published in: IEEE internet of things journal Vol. 10; no. 23; pp. 21039 - 21060
• Main Authors: Dai, Pengcheng, Liu, Hongzhe, Yu, Wenwu, Wang, He
• Format: Journal Article
• Language: English
• Published: Piscataway The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 01.12.2023
• Subjects: Algorithms; Approximation; Convergence; Linear functions; Machine learning; Mathematical analysis; Multiagent systems; Neural networks
• ISSN: 2327-4662; 2327-4662
• DOI: 10.1109/JIOT.2023.3284510

In this article, the fully distributed neural learning algorithms by neural network approximation for networked multiagent reinforcement learning (NMARL) are studied. To tackle the convergence analysis of methods in NMARL with tremendous state-action space, most of the existing distributed algorithms are designed by linear function approximation, which however would fall into a situation of poor expression. To conquer such limitation, the distributed neural learning algorithms are developed by using a novel neural network approximation that bridges the theory and practice of deep NMARL (DMARL). Specifically, inspired by the overparametrization method for minimizing mean-squared projected bellman error (MSPBE), the distributed neural learning algorithms with population semigradients and stochastic semigradients are respectively, proposed to solve the NMARL problem. Furthermore, the convergence of the proposed algorithms are strictly given by employing the overparametrization method to establish the approximate stationary point of MSPBE to characterize the algorithms toward the global optimum. Finally, some numerical simulations demonstrate the effectiveness of the distributed neural learning algorithms.