The devil is in the face: Exploiting harmonious representations for facial expression recognition

• Published in: Neurocomputing (Amsterdam) Vol. 486; pp. 104 - 113
• Main Authors: Han, Jiayi, Du, Liang, Ye, Xiaoqing, Zhang, Li, Feng, Jianfeng
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 14.05.2022
• Subjects: Facial expression recognition (FER); Feature learning; Graph convolution networks (GCNs); Graph convolution networks (GCNs); Feature learning; Facial expression recognition (FER)
• ISSN: 0925-2312
• DOI: 10.1016/j.neucom.2022.02.054

•It is important to learn spatial-invariant feature representation for facial expression recognition which introduces no extra cost during inference.•Landmark is helpful for recognizing facial expressions with GCNs.•Global feature is indispensable for producing discriminative features. Despite the recent effort from computer vision community, facial expression recognition (FER) remains a largely unsolved problem. This is because the appearance of people’s face undergoes dramatic changes due to changes in view angle, pose, illumination plus ambiguous facial expressions and low-quality facial images. In this work, we show the advantage of feature representation learning by dynamically graph message propagating subject to FER discriminative learning constraints and minimizing the distance of expression-agnostic transformed instance feature pairs. Specifically, we formulate a novel Harmonious Representation Learning (HRL) model for joint learning of landmark-guided graph message propagation, and spatially invariant feature learning using only generic matching metrics. Extensive comparative evaluations demonstrate the superiority of our proposed approach for FER over a variety of state-of-the-art methods on three major benchmark datasets including SFEW 2.0, RAF-DB, and CK+.