Toward Generalization of Automated Temporal Abstraction to Partially Observable Reinforcement Learning

• Published in: IEEE transactions on cybernetics Vol. 45; no. 8; pp. 1414 - 1425
• Main Authors: Cilden, Erkin, Polat, Faruk
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.08.2015
• Subjects: Approximation algorithms; Approximation methods; Entropy; History; Learning (artificial intelligence); Learning abstractions; Mathematical model; partially observable Markov decision process (POMDP); reinforcement learning (RL); Vectors; partially observable Markov decision process (POMDP); reinforcement learning (RL); Learning abstractions
• ISSN: 2168-2267; 2168-2275; 2168-2275
• DOI: 10.1109/TCYB.2014.2352038

Temporal abstraction for reinforcement learning (RL) aims to decrease learning time by making use of repeated sub-policy patterns in the learning task. Automatic extraction of abstractions during RL process is difficult but has many challenges such as dealing with the curse of dimensionality. Various studies have explored the subject under the assumption that the problem domain is fully observable by the learning agent. Learning abstractions for partially observable RL is a relatively less explored area. In this paper, we adapt an existing automatic abstraction method, namely extended sequence tree, originally designed for fully observable problems. The modified method covers a certain family of model-based partially observable RL settings. We also introduce belief state discretization methods that can be used with this new abstraction mechanism. The effectiveness of the proposed abstraction method is shown empirically by experimenting on well-known benchmark problems.