Fully Automatic Recognition of the Temporal Phases of Facial Actions

• Published in: IEEE transactions on systems, man and cybernetics. Part B, Cybernetics Vol. 42; no. 1; pp. 28 - 43
• Main Authors: Valstar, M. F., Pantic, M.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.02.2012
• Subjects: Artificial Intelligence; Cybernetics; Detectors; Emotion recognition; Emotions; Face; Face recognition; Facial; Facial Expression; Facial expression analysis; Filtering; GentleBoost; Gold; Hidden Markov models; Humans; Image Interpretation, Computer-Assisted - methods; Image sequences; particle filtering; Pattern Recognition, Automated - methods; Photography - methods; Recognition; spatiotemporal facial behavior analysis; Subtraction Technique; support vector machine (SVM); Temporal logic; Video Recording - methods
• ISSN: 1083-4419; 1941-0492; 1941-0492
• DOI: 10.1109/TSMCB.2011.2163710

Past work on automatic analysis of facial expressions has focused mostly on detecting prototypic expressions of basic emotions like happiness and anger. The method proposed here enables the detection of a much larger range of facial behavior by recognizing facial muscle actions [action units (AUs)] that compound expressions. AUs are agnostic, leaving the inference about conveyed intent to higher order decision making (e.g., emotion recognition). The proposed fully automatic method not only allows the recognition of 22 AUs but also explicitly models their temporal characteristics (i.e., sequences of temporal segments: neutral, onset, apex, and offset). To do so, it uses a facial point detector based on Gabor-feature-based boosted classifiers to automatically localize 20 facial fiducial points. These points are tracked through a sequence of images using a method called particle filtering with factorized likelihoods. To encode AUs and their temporal activation models based on the tracking data, it applies a combination of GentleBoost, support vector machines, and hidden Markov models. We attain an average AU recognition rate of 95.3% when tested on a benchmark set of deliberately displayed facial expressions and 72% when tested on spontaneous expressions.