RoCo: Dialectic Multi-Robot Collaboration with Large Language Models

• Published in: 2024 IEEE International Conference on Robotics and Automation (ICRA) pp. 286 - 299
• Main Authors: Mandi, Zhao, Jain, Shreeya, Song, Shuran
• Format: Conference Proceeding
• Language: English
• Published: IEEE 13.05.2024
• Subjects: Benchmark testing; Collaboration; Human in the loop; Large language models; Robot kinematics; Semantics; Trajectory planning
• DOI: 10.1109/ICRA57147.2024.10610855

We propose a novel approach to multi-robot collaboration that harnesses the power of pre-trained large language models (LLMs) for both high-level communication and low-level path planning. Robots are equipped with LLMs to discuss and collectively reason task strategies. They generate sub-task plans and task space waypoint paths, which are used by a multi-arm motion planner to accelerate trajectory planning. We also provide feedback from the environment, such as collision checking, and prompt the LLM agents to improve their plan and waypoints in-context. For evaluation, we introduce RoCoBench, a 6-task benchmark covering a wide range of multi-robot collaboration scenarios, accompanied by a text-only dataset that evaluates LLMs' agent representation and reasoning capability. We experimentally demonstrate the effectiveness of our approach - it achieves high success rates across all tasks in RoCoBench and adapts to variations in task semantics. Our dialog setup offers high interpretability and flexibility - in real world experiments, we show RoCo easily incorporates human-in-the-loop, where a user can communicate and collaborate with a robot agent to complete tasks together.