High-Quality Textured 3D Shape Reconstruction with Cascaded Fully Convolutional Networks

• Published in: IEEE transactions on visualization and computer graphics Vol. 27; no. 1; pp. 83 - 97
• Main Authors: Liu, Zheng-Ning, Cao, Yan-Pei, Kuang, Zheng-Fei, Kobbelt, Leif, Hu, Shi-Min
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: 3D vision; Algorithms; Angles (geometry); cascaded architecture; Color; Computer architecture; Datasets; Geometry; High-fidelity reconstruction; Image color analysis; Image reconstruction; Machine learning; Model accuracy; Neural networks; Noise measurement; Reconstruction; Shape; Solid modeling; Surface reconstruction; texture reconstruction; Three-dimensional displays
• ISSN: 1077-2626; 1941-0506; 1941-0506
• DOI: 10.1109/TVCG.2019.2937300

We present a learning-based approach to reconstructing high-resolution three-dimensional (3D) shapes with detailed geometry and high-fidelity textures. Albeit extensively studied, algorithms for 3D reconstruction from multi-view depth-and-color (RGB-D) scans are still prone to measurement noise and occlusions; limited scanning or capturing angles also often lead to incomplete reconstructions. Propelled by recent advances in 3D deep learning techniques, in this paper, we introduce a novel computation- and memory-efficient cascaded 3D convolutional network architecture, which learns to reconstruct implicit surface representations as well as the corresponding color information from noisy and imperfect RGB-D maps. The proposed 3D neural network performs reconstruction in a progressive and coarse-to-fine manner, achieving unprecedented output resolution and fidelity. Meanwhile, an algorithm for end-to-end training of the proposed cascaded structure is developed. We further introduce Human10 , a newly created dataset containing both detailed and textured full-body reconstructions as well as corresponding raw RGB-D scans of 10 subjects. Qualitative and quantitative experimental results on both synthetic and real-world datasets demonstrate that the presented approach outperforms existing state-of-the-art work regarding visual quality and accuracy of reconstructed models.