Nerfies: Deformable Neural Radiance Fields

• Published in: Proceedings / IEEE International Conference on Computer Vision pp. 5845 - 5854
• Main Authors: Park, Keunhong, Sinha, Utkarsh, Barron, Jonathan T., Bouaziz, Sofien, Goldman, Dan B, Seitz, Steven M., Martin-Brualla, Ricardo
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.10.2021
• Subjects: 3D from multiview and other sensors; Adaptation models; Computer vision; Deformable models; Geometry; Mobile handsets; Optimization methods; Rendering (computer graphics); Stereo
• ISSN: 2380-7504
• DOI: 10.1109/ICCV48922.2021.00581

We present the first method capable of photorealistically reconstructing deformable scenes using photos/videos captured casually from mobile phones. Our approach augments neural radiance fields (NeRF) by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone to local minima, and propose a coarse-to-fine optimization method for coordinate-based models that allows for more robust optimization. By adapting principles from geometry processing and physical simulation to NeRF-like models, we propose an elastic regularization of the deformation field that further improves robustness. We show that our method can turn casually captured selfie photos/videos into deformable NeRF models that allow for photorealistic renderings of the subject from arbitrary viewpoints, which we dub "nerfies." We evaluate our method by collecting time-synchronized data using a rig with two mobile phones, yielding train/validation images of the same pose at different viewpoints. We show that our method faithfully reconstructs non-rigidly deforming scenes and reproduces unseen views with high fidelity.