Hierarchical deep reinforcement learning to drag heavy objects by adult-sized humanoid robot

• Published in: Applied soft computing Vol. 110; p. 107601
• Main Authors: Saeedvand, Saeed, Mandala, Hanjaya, Baltes, Jacky
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.10.2021
• Subjects: Deep learning; Deep reinforcement learning; Dragging object; Humanoid robot; Deep learning; Deep reinforcement learning; Dragging object; Humanoid robot
• ISSN: 1568-4946; 1872-9681
• DOI: 10.1016/j.asoc.2021.107601

Most research on robot manipulation focuses on objects that are light enough for the robot to pick them up. However, in our daily life, some objects are too big or too heavy to be picked up or carried, so that dragging them is necessary. Although bipedal humanoid robots have nowadays good mobility on level ground, dragging unfamiliar objects including large and heavy objects on various surfaces is an interesting research area with many applications, which will provide insights into human manipulation and will encourage the development of novel algorithms for robot motion planning and control. This is a challenging problem, not only because of the unknown and potentially variable friction of the foot, but also because the feet of the robot may slip during unbalanced poses. In this paper, we propose a novel hierarchical deep learning algorithm that learns how to drag heavy objects with an adult-sized humanoid robot for the first time. First, we present a Three-layered Convolution Volumetric Network (TCVN) for 3D object classification with point clouds volumetric occupancy grid integration. Second, we propose a lightweight real-time instance segmentation method named Tiny-YOLACT for the detection and classification of the floor surface. Third, we propose a deep Q-learning algorithm to learn the policy control of the Center of Mass of the robot (DQL-COM). The DQL-COM algorithm learning is bootstrapped using the ROS Gazebo simulator. After initial training, we complete training on the THORMANG-Wolf, a 1.4 m tall adult-sized humanoid robot with 27 degrees of freedom and weighing 48 kg, on three distinct types of surfaces. We evaluate the performance of our approach by dragging eight different types of objects (e.g., a small suitcase, a large suitcase, a chair). The extensive experiments (480 times on the real robot) included dragging a heavy object with a mass of 84.6 kg (two times greater than the robot’s weight) and showed remarkable success rates of 92.92% when combined with the force–torque sensors, and 83.75% without force–torque sensors. •Proposing a hierarchical DQL algorithm for object dragging with adult-sized robot.•Proposing a Three-layered Convolution Volumetric Network (TCVN) model for 3D object detection.•Proposing a lightweight real-time instance segmentation algorithm named Tiny-YOLACT.•Simulating and implementing on a real robot with extensive experimental results.