Multimodal Emotion Recognition Based on Speech and Physiological Signals Using Deep Neural Networks

• Published in: Pattern Recognition. ICPR International Workshops and Challenges Vol. 12666; pp. 289 - 300
• Main Authors: Bakhshi, Ali, Chalup, Stephan
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2021; Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Deep neural network; Emotion recognition; Multimodal data
• ISBN: 9783030687793; 3030687791
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-030-68780-9_25

A suitable combination of data in a multimodal emotion recognition model allows conveying and combining each channel’s information to achieve a better recognition of the encoded emotion than would be possible using only a single modality and channel. In this paper, we focus on combining speech and physiological signals to predict the arousal and valence levels of the emotional states of a person. We designed a neural network that can use the information from raw audio signals, electrocardiograms, heart rate variability, electro-dermal activity, and skin conductance levels, to predict emotional states. The proposed deep neural network architecture works as an end-to-end process, which means, neither any pre-processing of the input data nor post-processing of the prediction of the network was applied. Using the data of the modalities available in the publicly accessible part of the RECOLA database, we achieved results comparable to other state-of-the-art approaches.