MoSAR: Monocular Semi-Supervised Model for Avatar Reconstruction using Differentiable Shading

• Published in: Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 1770 - 1780
• Main Authors: Dib, Abdallah, Gustavo Hafemann, Luiz, Got, Emeline, Anderson, Trevor, Fadaeinejad, Amin, Cruz, Rafael M. O., Carbonneau, Marc-Andre
• Format: Conference Proceeding
• Language: English
• Published: IEEE 16.06.2024
• Subjects: Avatars; Differentiable Rendering; Face recognition; Face Reconstruction; Geometry; Reflectance Estimation; Reflectivity; Rendering (computer graphics); Three-dimensional displays; Training
• ISSN: 1063-6919
• DOI: 10.1109/CVPR52733.2024.00174

Reconstructing an avatar from a portrait image has many applications in multimedia, but remains a challenging research problem. Extracting reflectance maps and geom- etry from one image is ill-posed: recovering geometry is a one-to-many mapping problem and reflectance and light are difficult to disentangle. Accurate geometry and reflectance can be captured under the controlled conditions of a light stage, but it is costly to acquire large datasets in this fash- ion. Moreover, training solely with this type of data leads to poor generalization with in-the-wild images. This moti- vates the introduction of MoSAR, a method for 3D avatar generation from monocular images. We propose a semi- supervised training scheme that improves generalization by learning from both light stage and in-the-wild datasets. This is achieved using a novel differentiable shading formulation. We show that our approach effectively disentangles the intrinsic face parameters, producing relightable avatars. As a result, MoSAR 1 1 Project page: https://ubisoft-laforge.github.io/character/mosar estimates a richer set of skin reflectance maps and generates more realistic avatars than existing state-of-the-art methods. We also release a new dataset, that provides intrinsic face attributes (diffuse, specular, am- bient occlusion and translucency maps) for 10k subjects.