A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference

• Published in: IEEE transactions on multimedia Vol. 12; no. 7; pp. 682 - 691
• Main Authors: Wang, Shangfei, Liu, Zhilei, Lv, Siliang, Lv, Yanpeng, Wu, Guobing, Peng, Peng, Chen, Fei, Wang, Xufa
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.11.2010; Institute of Electrical and Electronics Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Active appearance model; Applied sciences; Artificial intelligence; Cameras; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Discriminant analysis; Emotion inference; Emotion recognition; Emotions; Exact sciences and technology; expression recognition; Face recognition; Facial; facial expression; Illumination; Image databases; Image sequences; Inference; Infrared; infrared image; Lighting; Linear discriminant analysis; Pattern recognition. Digital image processing. Computational geometry; Principal component analysis; Recognition; Software; Spontaneous; spontaneous database; Studies; Timing; visible image; Infrared camera; Model matching; Image matching; Facies; visible image; Database; Illumination; Visible spectrum; expression recognition; Multimedia; Infrared thermography; Computer vision; Discriminant analysis; Statistical analysis; Face recognition; Inference; Emotion emotionality; Timed system; Emotion inference; infrared image; Luminance; spontaneous database; Thermal imaging; Usability; Facial expression; Principal component analysis
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2010.2060716

To date, most facial expression analysis has been based on visible and posed expression databases. Visible images, however, are easily affected by illumination variations, while posed expressions differ in appearance and timing from natural ones. In this paper, we propose and establish a natural visible and infrared facial expression database, which contains both spontaneous and posed expressions of more than 100 subjects, recorded simultaneously by a visible and an infrared thermal camera, with illumination provided from three different directions. The posed database includes the apex expressional images with and without glasses. As an elementary assessment of the usability of our spontaneous database for expression recognition and emotion inference, we conduct visible facial expression recognition using four typical methods, including the eigenface approach [principle component analysis (PCA)], the fisherface approach [PCA + linear discriminant analysis (LDA)], the Active Appearance Model (AAM), and the AAM-based + LDA. We also use PCA and PCA+LDA to recognize expressions from infrared thermal images. In addition, we analyze the relationship between facial temperature and emotion through statistical analysis. Our database is available for research purposes.