GraspNeRF: Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRF

• Published in: 2023 IEEE International Conference on Robotics and Automation (ICRA) pp. 1757 - 1763
• Main Authors: Dai, Qiyu, Zhu, Yan, Geng, Yiran, Ruan, Ciyu, Zhang, Jiazhao, Wang, He
• Format: Conference Proceeding
• Language: English
• Published: IEEE 29.05.2023
• Subjects: 6-DOF; Geometry; Grasping; Real-time systems; Robot vision systems; Sensors; Training data
• DOI: 10.1109/ICRA48891.2023.10160842

In this work, we tackle 6-DoF grasp detection for transparent and specular objects, which is an important yet challenging problem in vision-based robotic systems, due to the failure of depth cameras in sensing their geometry. We, for the first time, propose a multiview RGB-based 6-DoF grasp detection network, GraspNeRF, that leverages the generalizable neural radiance field (NeRF) to achieve material-agnostic object grasping in clutter. Compared to the existing NeRF-based 3-DoF grasp detection methods that rely on densely captured input images and time-consuming per-scene optimization, our system can perform zero-shot NeRF construction with sparse RGB inputs and reliably detect 6-DoF grasps, both in real-time. The proposed framework jointly learns generalizable NeRF and grasp detection in an end-to-end manner, optimizing the scene representation construction for the grasping. For training data, we generate a large-scale photorealistic domain-randomized synthetic dataset of grasping in cluttered tabletop scenes that enables direct transfer to the real world. Our extensive experiments in synthetic and real-world environments demonstrate that our method significantly outperforms all the baselines in all the experiments while remaining in real-time. Project page can be found at https://pku-epic.github.io/GraspNeRF.