GNN-Based Embedded Framework for Consumer Affect Recognition Using Thermal Facial ROIs

• Published in: IEEE consumer electronics magazine Vol. 12; no. 4; pp. 74 - 83
• Main Authors: Nayak, Satyajit, Routray, Aurobinda, Sarma, Monalisa, Uttarkabat, Satarupa
• Format: Magazine Article
• Language: English
• Published: Piscataway IEEE 01.07.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Affect Recognition; Algorithms; Arousal; Computer architecture; Consumer electronics; Deep learning; Embedded GNN; Face recognition; Faces; Faster R-CNN; Feature recognition; Graph neural networks; Hardware; Head movement; k-NN; Machine learning; MIL; MTS; Nose; Real time; Real-time systems; Remote monitoring; Skin temperature; Thermal Facial ROIs; Training; Visualization
• ISSN: 2162-2248; 2162-2256
• DOI: 10.1109/MCE.2022.3153748

Recently consumer electronics products for the human device or machine interaction in smart healthcare systems have been widespread due to the progress in health monitoring hardware and remote diagnostic services. Affect recognition through thermal facial signatures is significant in real-time human–machine interaction (HMI) studies. The distribution of facial skin temperature displays explicit characteristics related to affect arousal. When human interacts with a machine or computer, it is challenging to detect face and track the facial regions of interest (ROIs) in a thermal video because of head motion artifacts. Our proposed embedded portable HMI product integrated with the required hardware and deep learning architecture is used to recognize the human affect in real time. The transfer learning approach (faster R-CNN) and the multiple instance learning algorithm are applied to thermal video for thermal face and ROIs detection and tracking. The multivariate time series (MTS) features generated from specific ROIs’ calculated mean intensity variations.This study proposes a k-nearest neighbor algorithm-based graph neural network architecture that utilizes MTS features to recognize affects in the facial thermal video. The effectiveness of the proposed algorithm for the in-house and NVIE thermal dataset gives better accuracy of 76.05% and 71.91% compared with state-of-the-art approaches.