Deep Learning Based Emotion Recognition with the Use of Time-Scale Two-Dimensional Transformation from Electrocardiogram Signals

• Published in: THE TRANSACTION OF THE KOREAN INSTITUTE OF ELECTRICAL ENGINEERS P Vol. 70P; no. 3; pp. 163 - 173
• Main Authors: Lee, Myung-Won, Byeon, Yeong-Hyeon, Yeom, Chan-Uk, Kwak, Keun-Chang
• Format: Journal Article
• Language: English
• Published: 대한전기학회 30.09.2021
• Subjects: 전기공학; Ensemble Deep Models; ASCERTAIN Sentiment Database; Two-Dimensional Time-Scale Representation; Electrocardiogram; Emotion Recognition
• ISSN: 1229-800X; 2586-7792
• DOI: 10.5370/KIEE.2021.70.3.163

This paper describes the design of ensemble deep models based on time-scale transformation from electrocardiogram (ECG) signals for emotion recognition. As the number of senior citizens living alone increases, emotion robots that can interact with them and other emotion robots are becoming increasingly important. Existing emotion robots usually recognized emotion through images of the user’s facial expressions or voice signals. However, there are many situations where the user’s emotions cannot read under various environments. Therefore, research on recognizing the user’s emotions through ECG signals among different biomedical signals is actively being conducted. The proposed method converts ECG signals into various types of two-dimensional time-scale representations. We then designed a four-stream deep learning model by applying it to an ensemble form and transfer learning. Finally, an experiment was conducted using the ASCERTAIN sentiment database. This database contains data recorded by 58 people with 9 different emotions. Among these emotions, we used six representatives (surprise, happiness, anger, disgust, fear, and sadness). The experimental results revealed that the presented ensemble deep models showed good performance in comparison with each single deep model and the original model without transformation KCI Citation Count: 0