Enhancing Recognition of Stereotyped Movements in ASD Children Through Action Pattern Mining and Multi-Channel Fusion

• Published in: IEEE journal of biomedical and health informatics Vol. 29; no. 3; pp. 2020 - 2033
• Main Authors: Zhang, Baiqiao, Yuan, Yanran, Qin, Wei, Li, Xiangxian, Liu, Weiying, Yao, Wenxin, Bian, Yulong, Liu, Juan
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.03.2025
• Subjects: Accuracy; Algorithms; Autism; Autism spectrum disorder; Autism Spectrum Disorder - diagnosis; Autism Spectrum Disorder - diagnostic imaging; Autism Spectrum Disorder - physiopathology; Bioinformatics; Child; Child, Preschool; Data mining; Feature extraction; Female; Human activity recognition; Humans; Image Interpretation, Computer-Assisted - methods; Male; Movement - physiology; Multi-channel fusion; Optical flow; Pattern recognition; Skeleton; Stereotyped Behavior - physiology; Stereotyped movement; Videos; Visualization
• ISSN: 2168-2194; 2168-2208; 2168-2208
• DOI: 10.1109/JBHI.2024.3511601

Stereotyped movements play a crucial role in diagnosing Autism Spectrum Disorder (ASD). However, recognizing them poses challenges, due to limited data availability and the movements' specificity and varying duration. To support in-depth analysis of ASD children's movements, we constructed the ACSA653 dataset, comprising 653 videos across six classes of stereotyped movements. This dataset surpasses existing ones in both scale and category. To improve the recognition of stereotyped movements, we propose APMFNet, a model that integrates three modules: Visual Motion Learning (VML), Skeleton Relation Mining (SRM), and Multi-channel Fusion (MF). The VML module focuses on extracting spatial and motion information from RGB and optical-flow sequences. The SRM module effectively mines essential motion patterns associated with stereotyped movements through cross-modal graph. The MF module fuses multi-modal information through cross-modality attention to facilitate decision-making. Tested on ACSA653, APMFNet outperforms current state-of-the-art methods, suggesting its potential to identify stable patterns of stereotyped movements in children with ASD.