Code3 A System for End-to-End Programming of Mobile Manipulator Robots for Novices and Experts

• Published in: 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI pp. 453 - 462
• Main Authors: Huang, Justin, Cakmak, Maya
• Format: Conference Proceeding
• Language: English
• Published: New York, NY, USA ACM 06.03.2017
• Series: ACM Conferences
• Subjects: Computing methodologies > Artificial intelligence > Computer vision > Computer vision tasks > Vision for robotics; Educational robots; End-user programming; Manipulators; Mobile Manipulation; Programming; Programming by demonstration; Programming tools; Robot sensing systems; Software and its engineering > Software notations and tools > Context specific languages > Programming by example; Software and its engineering > Software notations and tools > Context specific languages > Visual languages; Task analysis; Three-dimensional displays; mobile manipulation; programming tools; end-user programming; programming by demonstration
• ISBN: 9781450343367; 1450343368
• ISSN: 2167-2148
• DOI: 10.1145/2909824.3020215

This paper introduces Code3, a system for user-friendly, rapid programming of mobile manipulator robots. The system is designed to let non-roboticists and roboticists alike program end-to-end manipulation tasks. To accomplish this, Code3 provides three integrated components for perception, manipulation, and high-level programming. The perception component helps users define a library of object and scene parts that the robot can later detect. The manipulation component lets users define actions for manipulating objects or scene parts through programming by demonstration. Finally, the high-level programming component provides a drag-and-drop interface with which users can program the logic and control flow to accomplish a task using their previously specified perception and manipulation capabilities. We present findings from an observational user study with non-roboticist programmers (N=10) that demonstrate their ability to quickly learn Code3 and program a PR2 robot to do manipulation tasks. We also demonstrate how the system is expressive enough for an expert to rapidly program highly complex manipulation tasks like playing tic-tac-toe and reconfiguring an object to be graspable.