Modeling, tracking and interactive animation of faces and heads//using input from video

• Published in: Proceedings Computer Animation '96 pp. 68 - 79
• Main Authors: Essa, I., Basu, S., Darrell, T., Pentland, A.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 1996
• Subjects: Displacement control; Displacement measurement; Facial animation; Geometry; Magnetic heads; Motion control; Motion measurement; Muscles; Solid modeling; Video sharing
• ISBN: 9780818675881; 0818675888
• ISSN: 1087-4844
• DOI: 10.1109/CA.1996.540489

We describe tools that use measurements from video for the extraction of facial modeling and animation parameters, head tracking, and real time interactive facial animation. These tools share common goals but rely on varying details of physical and geometric modeling and in their input measurement system. Accurate facial modeling involves fine details of geometry and muscle coarticulation. By coupling pixel by pixel measurements of surface motion to a physically based face model and a muscle control model, we have been able to obtain detailed spatio temporal records of both the displacement of each point on the facial surface and the muscle control required to produce the observed facial motion. We discuss the importance of this visually extracted representation in terms of realistic facial motion synthesis. A similar method that uses an ellipsoidal model of the head coupled with detailed estimates of visual motion allows accurate tracking of head motion in 3D. Additionally, by coupling sparse, fast visual measurements with our physically based model via an interpolation process, we have produced a real time interactive facial animation/mimicking system.