Permutation Invariant Agent-Specific Centralized Critic in Multi-Agent Reinforcement Learning - Noppakun, Patsornchai; Akkarajitsakul, Khajonpong (2022 6th International Conference on Information Technology (InCIT), 2022)
We proposed a permutation invariant agent-specific centralized critic using graph convolutional networks in multiagent reinforcement learning. We consider an environment with partial observability where a joint observation of homogeneous agents is used as a state information in centralized training. A joint observation of homogeneous agents is permutation invariant, meaning that different permutations must be treated as the same. However, a traditional deep network like multilayer perceptron (MLP) outputs different values to different permutations, despite being the same data. A centralized critic using MLPs to represent joint observation of homogeneous agents suffers from data inefficiency because it only learns a single permutation instead of all permutations. Previous work has addressed this problem using graph convolutional networks (GCN) for "agent-agnostic'' centralized critics. Our work extends the use of GCNs to an "agent-specific'' centralized critic such as the critic used in Counterfactual Multi-Agent Policy Gradients (COMA) algorithm. We introduce three GCN variants of agentspecific critic architectures. Our experimental results on the multi-agent particle environment with COMA algorithm show that all GCN critics outperform the MLP baseline critics. Finally, we concluded that as the number of agents increases, the critic that takes advantage of agent homogeneity by separating global and local feature representation is the most scalable in terms of time complexity.