A deep learning model for classifying human facial expressions from infrared thermal images

• Published in: Scientific reports Vol. 11; no. 1; pp. 20696 - 17
• Main Authors: Bhattacharyya, Ankan, Chatterjee, Somnath, Sen, Shibaprasad, Sinitca, Aleksandr, Kaplun, Dmitrii, Sarkar, Ram
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 19.10.2021; Nature Publishing Group; Nature Portfolio
• Subjects: 631/1647/245; 631/61/185; Cameras; Cognition - physiology; Cognitive ability; Deep Learning; Disease control; Emotions - physiology; Facial Expression; Facial Recognition - physiology; Female; Humanities and Social Sciences; Humans; multidisciplinary; Pattern recognition; Science; Science (multidisciplinary); Spectrophotometry, Infrared - methods
• ISSN: 2045-2322; 2045-2322
• DOI: 10.1038/s41598-021-99998-z

The analysis of human facial expressions from the thermal images captured by the Infrared Thermal Imaging (IRTI) cameras has recently gained importance compared to images captured by the standard cameras using light having a wavelength in the visible spectrum. It is because infrared cameras work well in low-light conditions and also infrared spectrum captures thermal distribution that is very useful for building systems like Robot interaction systems, quantifying the cognitive responses from facial expressions, disease control, etc. In this paper, a deep learning model called IRFacExNet ( I nfra R ed Fac ial Ex pression Net work) has been proposed for facial expression recognition (FER) from infrared images. It utilizes two building blocks namely Residual unit and Transformation unit which extract dominant features from the input images specific to the expressions. The extracted features help to detect the emotion of the subjects in consideration accurately. The Snapshot ensemble technique is adopted with a Cosine annealing learning rate scheduler to improve the overall performance. The performance of the proposed model has been evaluated on a publicly available dataset, namely IRDatabase developed by RWTH Aachen University. The facial expressions present in the dataset are Fear, Anger, Contempt, Disgust, Happy, Neutral, Sad, and Surprise. The proposed model produces 88.43% recognition accuracy, better than some state-of-the-art methods considered here for comparison. Our model provides a robust framework for the detection of accurate expression in the absence of visible light.