Neural Rendering for Game Character Auto-Creation

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 44; no. 3; pp. 1489 - 1502
• Main Authors: Shi, Tianyang, Zou, Zhengxia, Shi, Zhenwei, Yuan, Yi
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.03.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; deep learning; Engines; Face recognition; Faces; Game character customization; Games; Image reconstruction; Mesh generation; neural rendering; Parameters; Rendering; Rendering (computer graphics); role-playing games; Solid modeling; Three-dimensional displays; Video Games
• ISSN: 0162-8828; 1939-3539; 2160-9292; 1939-3539
• DOI: 10.1109/TPAMI.2020.3024009

Many role-playing games feature character creation systems where players are allowed to edit the facial appearance of their in-game characters. This paper proposes a novel method to automatically create game characters based on a single face photo. We frame this "artistic creation" process under a self-supervised learning paradigm by leveraging the differentiable neural rendering. Considering the rendering process of a typical game engine is not differentiable, an "imitator" network is introduced to imitate the behavior of the engine so that the in-game characters can be smoothly optimized by gradient descent in an end-to-end fashion. Different from previous monocular 3D face reconstruction which focuses on generating 3D mesh-grid and ignores user interaction, our method produces fine-grained facial parameters with a clear physical significance where users can optionally fine-tune their auto-created characters by manually adjusting those parameters. Experiments on multiple large-scale face datasets show that our method can generate highly robust and vivid game characters. Our method has been applied to two games and has now provided over 10 million times of online services.