On-Road Driver Emotion Recognition Using Facial Expression

• Published in: Applied sciences Vol. 12; no. 2; p. 807
• Main Authors: Xiao, Huafei, Li, Wenbo, Zeng, Guanzhong, Wu, Yingzhang, Xue, Jiyong, Zhang, Juncheng, Li, Chengmou, Guo, Gang
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.01.2022
• Subjects: Accuracy; affective computing; Classification; Datasets; Deep learning; driving safety; Emotions; Experiments; facial expression recognition; human-machine interaction; Machine learning; Neural networks; Road rage; Roads & highways; smart cockpit; Speech; Traffic accidents & safety; Traffic congestion
• ISSN: 2076-3417; 2076-3417
• DOI: 10.3390/app12020807

With the development of intelligent automotive human-machine systems, driver emotion detection and recognition has become an emerging research topic. Facial expression-based emotion recognition approaches have achieved outstanding results on laboratory-controlled data. However, these studies cannot represent the environment of real driving situations. In order to address this, this paper proposes a facial expression-based on-road driver emotion recognition network called FERDERnet. This method divides the on-road driver facial expression recognition task into three modules: a face detection module that detects the driver’s face, an augmentation-based resampling module that performs data augmentation and resampling, and an emotion recognition module that adopts a deep convolutional neural network pre-trained on FER and CK+ datasets and then fine-tuned as a backbone for driver emotion recognition. This method adopts five different backbone networks as well as an ensemble method. Furthermore, to evaluate the proposed method, this paper collected an on-road driver facial expression dataset, which contains various road scenarios and the corresponding driver’s facial expression during the driving task. Experiments were performed on the on-road driver facial expression dataset that this paper collected. Based on efficiency and accuracy, the proposed FERDERnet with Xception backbone was effective in identifying on-road driver facial expressions and obtained superior performance compared to the baseline networks and some state-of-the-art networks.