Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks

• Published in: Neural networks Vol. 129; pp. 149 - 162
• Main Authors: Han, Dongqi, Doya, Kenji, Tani, Jun
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.09.2020
• Subjects: Compositionality; Multiple timescale; Partially observable Markov decision process; Recurrent neural network; Reinforcement learning; Multiple timescale; Recurrent neural network; Partially observable Markov decision process; Compositionality; Reinforcement learning
• ISSN: 0893-6080; 1879-2782; 1879-2782
• DOI: 10.1016/j.neunet.2020.06.002

Recurrent neural networks (RNNs) for reinforcement learning (RL) have shown distinct advantages, e.g., solving memory-dependent tasks and meta-learning. However, little effort has been spent on improving RNN architectures and on understanding the underlying neural mechanisms for performance gain. In this paper, we propose a novel, multiple-timescale, stochastic RNN for RL. Empirical results show that the network can autonomously learn to abstract sub-goals and can self-develop an action hierarchy using internal dynamics in a challenging continuous control task. Furthermore, we show that the self-developed compositionality of the network enhances faster re-learning when adapting to a new task that is a re-composition of previously learned sub-goals, than when starting from scratch. We also found that improved performance can be achieved when neural activities are subject to stochastic rather than deterministic dynamics.