Hierarchical reinforcement learning with movement primitives

• Published in: 2011 11th IEEE-RAS International Conference on Humanoid Robots pp. 231 - 238
• Main Authors: Stulp, Freek, Schaal, Stefan
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.10.2011
• Subjects: Aerospace electronics; Cost function; Learning; Noise; Robots; Shape; Trajectory
• ISBN: 9781612848662; 1612848664
• ISSN: 2164-0572
• DOI: 10.1109/Humanoids.2011.6100841

Temporal abstraction and task decomposition drastically reduce the search space for planning and control, and are fundamental to making complex tasks amenable to learning. In the context of reinforcement learning, temporal abstractions are studied within the paradigm of hierarchical reinforcement learning. We propose a hierarchical reinforcement learning approach by applying our algorithm PI 2 to sequences of Dynamic Movement Primitives. For robots, this representation has some important advantages over discrete representations in terms of scalability and convergence speed. The parameters of the Dynamic Movement Primitives are learned simultaneously at different levels of temporal abstraction. The shape of a movement primitive is optimized w.r.t. the costs up to the next primitive in the sequence, and the subgoals between two movement primitives w.r.t. the costs up to the end of the entire movement primitive sequence. We implement our approach on an 11-DOF arm and hand, and evaluate it in a pick-and-place task in which the robot transports an object between different shelves in a cupboard.