Interrelation Between Speech and Facial Gestures in Emotional Utterances: A Single Subject Study

• Published in: IEEE transactions on audio, speech, and language processing Vol. 15; no. 8; pp. 2331 - 2347
• Main Authors: Busso, C., Narayanan, S.S.
• Format: Journal Article
• Language: English
• Published: Piscataway, NJ IEEE 01.11.2007; Institute of Electrical and Electronics Engineers
• Subjects: Acoustics; Affective state; Applied sciences; articulatory movements; Audio databases; Channels; Communication channels; Correlation; Emotion recognition; Emotions; Exact sciences and technology; Eyebrows; Facial; Facial animation; Facial features; facial motion; Humans; Image processing; Information, signal and communications theory; Mapping; Principal component analysis; Sentences; Signal processing; Spatial databases; Speech; speech acoustic; Telecommunications and information theory; speech acoustic; Audio databases; Sound recording; Emotion recognition; Acoustic signal; Image processing; Semantic analysis; Transmission channel; Subspace method; Mapping; articulatory movements; Statistical method; facial motion; Consonant; Prosody; Audio signal; Image synthesis; Gesture; Phonetics; Affective state; Facial expression; Principal component analysis
• ISSN: 1558-7916
• DOI: 10.1109/TASL.2007.905145

The verbal and nonverbal channels of human communication are internally and intricately connected. As a result, gestures and speech present high levels of correlation and coordination. This relationship is greatly affected by the linguistic and emotional content of the message. The present paper investigates the influence of articulation and emotions on the interrelation between facial gestures and speech. The analyses are based on an audio-visual database recorded from an actress with markers attached to her face, who was asked to read semantically neutral sentences, expressing four emotion states (neutral, sadness, happiness, and anger). A multilinear regression framework is used to estimate facial features from acoustic speech parameters. The levels of coupling between the communication channels are quantified by using Pearson's correlation between the recorded and estimated facial features. The results show that facial and acoustic features are strongly interrelated, showing levels of correlation higher than r = 0.8 when the mapping is computed at sentence-level using spectral envelope speech features. The results reveal that the lower face region provides the highest activeness and correlation levels. Furthermore, the correlation levels present significant interemo- tional differences, which suggest that emotional content affect the relationship between facial gestures and speech. Principal component analysis (PCA) shows that the audiovisual mapping parameters are grouped in a smaller subspace, which suggests that there is an emotion-dependent structure that is preserved from across sentences. The results suggest that this internal structure seems to be easy to model when prosodic-features are used to estimate the audiovisual mapping. The results also reveal that the correlation levels within a sentence vary according to broad phonetic properties presented in the sentence. Consonants, especially unvoiced and fricative sounds, present the lowest correlation levels. Likewise, the results show that facial gestures are linked at different resolutions. While the orofacial area is locally connected with the speech, other facial gestures such as eyebrow motion are linked only at the sentence-level. The results presented here have important implications for applications such as facial animation and multimodal emotion recognition.