Fusion of forehead EEG with machine vision for real-time fatigue detection in an automatic processing pipeline

• Published in: Neural computing & applications Vol. 35; no. 12; pp. 8859 - 8872
• Main Authors: Min, Jianliang, Cai, Ming, Gou, Chao, Xiong, Chen, Yao, Xuejiao
• Format: Journal Article
• Language: English
• Published: London Springer London 01.04.2023; Springer Nature B.V
• Subjects: Artificial Intelligence; Computational Biology/Bioinformatics; Computational Science and Engineering; Computer Science; Data integration; Data Mining and Knowledge Discovery; Driver fatigue; Electroencephalography; Eye movements; Forehead; Image Processing and Computer Vision; Machine vision; Probability and Statistics in Computer Science; S.I.: AI based Techniques and Applications for Intelligent IoT Systems; Special Issue on Artificial Intelligence based Techniques and Applications for Intelligent IoT Systems (AI-TAIoT); Temporal resolution; Traffic accidents; Vision systems; SIFT; Forehead EEG; Machine vision; Driving fatigue; Eye movement; Multimodal data fusion
• ISSN: 0941-0643; 1433-3058
• DOI: 10.1007/s00521-022-07466-0

Driving fatigue is a leading contributor to traffic accidents and fatalities. For automatic detection of fatigue, multimodal data fusion is a potential key technique, especially the merging of electroencephalogram (EEG) and eye movement. Although EEG can produce objective measures of fatigue with very high temporal resolution, an unfriendly multichannel system limits its practical application while portable wearables and machine vision receive much attention since they are pervasive and user friendly. Hence, this study aims to construct a novel pipeline by using machine vision technique to improve the quality of driver fatigue detection based on forehead EEG. By coupling the Karolinska Sleepiness Scale (KSS) and percentage of eyelid closure (PERCLOS), the precise and reliable dataset for reflecting drivers’ fatigue levels were obtained. Moreover, major artifact contamination related to blink activity for frontal-channel EEG was removed by a synchronous video-based eyeblink event marker. In addition, the scale-invariant feature transform (SIFT) features of eyelid keypoints was applied to fuse with the EEG-driven features. Sixteen subjects participated in a realistic driving simulation experiment. Both machine and deep learning methods were used to implement the intra-subject and inter-subject cross validation. It demonstrated that our proposed method can achieve a significant performance. The present work showed the usefulness of forehead EEG and eye images that was originally merged for detecting fatigue. It provides a new strategy of using forehead EEG combined with machine vision to design a potential and automatic pipeline for driver fatigue detection.