Dual-stream network with cross-layer attention and similarity constraint for micro-expression recognition

• Published in: Multimedia systems Vol. 30; no. 3
• Main Authors: Wang, Gang, Huang, Shucheng
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.06.2024; Springer Nature B.V
• Subjects: Computer Communication Networks; Computer Graphics; Computer Science; Cryptology; Data Storage Representation; Flow mapping; Multimedia Information Systems; Operating Systems; Optical flow (image analysis); Recognition; Regular Paper; Representations; Similarity; Emotion recognition; Cross-layer attention; Micro-expression recognition; Similarity constraint
• ISSN: 0942-4962; 1432-1882
• DOI: 10.1007/s00530-024-01352-6

Micro-expression recognition (MER) is a pivotal research area within human emotion analysis. However, the fleeting, subtle, and complex nature of micro-expressions poses challenges for accurate and efficient recognition. To address this, this paper proposes a Dual-Stream Network with Cross-layer Attention and Similarity Constraint (DSN-CASC) for MER. The network is designed with two parallel branches, each dedicated to learning features of stacked optical flow maps and independent micro-expression optical flow components. The network amplifies its focus on representing features at different hierarchical levels via incorporating a cross-layer attention module, thereby enhancing the capture of key features associated with micro-expression variations. Furthermore, a similarity constraint strategy is introduced to ensure that the micro-expression features extracted by the dual branches exhibit similar representations, improving the network’s overall representation capability. Finally, a simple feature fusion approach is employed for micro-expression classification. Extensive experiments on the composite database validate the effectiveness of DSN-CASC under leave-one-subject-out cross-validation and composite database evaluation protocol. The results demonstrate that our proposed approach achieves promising performance improvements, which provide new insights and effective solutions for MER research.