Grounding Language with Visual Affordances over Unstructured Data

• Published in: 2023 IEEE International Conference on Robotics and Automation (ICRA) pp. 11576 - 11582
• Main Authors: Mees, Oier, Borja-Diaz, Jessica, Burgard, Wolfram
• Format: Conference Proceeding
• Language: English
• Published: IEEE 29.05.2023
• Subjects: Affordances; Data collection; Data models; Grounding; Multitasking; Natural languages; Visualization
• DOI: 10.1109/ICRA48891.2023.10160396

Recent works have shown that Large Language Models (LLMs) can be applied to ground natural language to a wide variety of robot skills. However, in practice, learning multi-task, language-conditioned robotic skills typically requires large-scale data collection and frequent human intervention to reset the environment or help correcting the current policies. In this work, we propose a novel approach to efficiently learn general-purpose language-conditioned robot skills from unstructured, offline and reset-free data in the real world by exploiting a self-supervised visuo-lingual affordance model, which requires annotating as little as 1% of the total data with language. We evaluate our method in extensive experiments both in simulated and real-world robotic tasks, achieving state-of-the-art performance on the challenging CALVIN benchmark and learning over 25 distinct visuomotor manipulation tasks with a single policy in the real world. We find that when paired with LLMs to break down abstract natural language instructions into subgoals via few-shot prompting, our method is capable of completing long-horizon, multi-tier tasks in the real world, while requiring an order of magnitude less data than previous approaches. Code and videos are available at http://hulc2.cs.uni-freiburg.de.