An LLM-based vision and language cobot navigation approach for Human-centric Smart Manufacturing

• Published in: Journal of manufacturing systems Vol. 75; pp. 299 - 305
• Main Authors: Wang, Tian, Fan, Junming, Zheng, Pai
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.08.2024
• Subjects: Human-centric smart manufacturing; Large language model; Vision and language navigation; Vision and language navigation; Human-centric smart manufacturing; Large language model
• ISSN: 0278-6125; 1878-6642
• DOI: 10.1016/j.jmsy.2024.04.020

Industry 5.0 prioritizes Human-centric Smart Manufacturing (HSM), aiming to enhance human operators’ well-being and needs. This necessitates collaborative robots with advanced natural interaction capabilities and improved perception, cognition, and action intelligence. The Large Language Model (LLM) exhibits strong reasoning abilities and generalization capabilities, which can significantly advance the development of HSM once integrated into human–robot interaction and collaboration. Accordingly, this paper explores part of LLM’s ability in the context of smart manufacturing, focusing on addressing interruptions in the manufacturing process caused by repetitive tool fetching. To alleviate this issue, a vision and language cobot navigation approach is innovatively adopted in the manufacturing environment that could be further used to assist operators in retrieving tools. Specifically, a real Human–Robot Collaboration (HRC) manufacturing scene is first reconstructed and annotated using Three-Dimensional (3D) point cloud techniques. Then the LLM is utilized for the Automated Guided Vehicle (AGV) to comprehend natural language commands and generate Python code to triger navigation actions. Finally, the Pathfinder algorithm is applied for corresponding path planning. The framework is implemented in the Artificial Intelligence (AI) Habitat simulator, and the case studies demonstrate that the AGV can accurately comprehend complex language instructions, empowering human operators to complete manufacturing tasks efficiently.