3DGAM: using 3D gesture and CAD models for training on mixed reality remote collaboration

• Published in: Multimedia tools and applications Vol. 80; no. 20; pp. 31059 - 31084
• Main Authors: Wang, Peng, Bai, Xiaoliang, Billinghurst, Mark, Zhang, Shusheng, Wei, Sili, Xu, Guangyao, He, Weiping, Zhang, Xiangyu, Zhang, Jie
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.08.2021
• Subjects: Computer Communication Networks; Computer Science; Data Structures and Information Theory; Multimedia Information Systems; Special Purpose and Application-Based Systems; Remote collaboration; Sharing gesture; Physical tasks; Mixed reality; Augmented reality; 3D CAD models
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-020-09731-7

As Virtual Reality(VR), Augmented Reality(AR), Mixed Reality(MR) technology becomes more accessible, it is important to explore VR/AR/MR technologies that can be used for remote collaboration on physical tasks. Previous research has shown that gesture-based interaction is intuitive and expressive for remote collaboration, and using 3D CAD models can provide clear instructions for assembly tasks. In this paper, therefore, we describe a new MR remote collaboration system which combines the use of gesture and CAD models in a complementary manner. The prototype system enables a remote expert in VR to provide instructions based on 3D gesture and CAD models (3DGAM) for a local worker who uses AR to see these instructions. Using this interface, we conducted a formal user study to explore the effect of sharing 3D gesture and CAD models in an assembly training task. We found that the combination of 3D gesture and CAD models can improve remote collaboration on an assembly task with respect to the performance time and user experience. Finally, we provide some conclusions and directions for future research.