Cross-Subject and Cross-Modal Transfer for Generalized Abnormal Gait Pattern Recognition

• Published in: IEEE transaction on neural networks and learning systems Vol. 32; no. 2; pp. 546 - 560
• Main Authors: Gu, Xiao, Guo, Yao, Deligianni, Fani, Lo, Benny, Yang, Guang-Zhong
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.02.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Abnormalities; Algorithms; Biomechanical Phenomena; Biometry; Body sensor network; Computer Systems; Data mining; Deep Learning; Electromyography; Feature extraction; Feature recognition; Gait; gait analysis; Gait Disorders, Neurologic - diagnosis; Gait recognition; Health care; Home Environment; Humans; Imaging, Three-Dimensional; Joints - diagnostic imaging; Kinematics; Lower Extremity - diagnostic imaging; model generalization; Motion capture; multimodal representation; Neural Networks, Computer; New technology; Noise measurement; Pattern recognition; Pattern Recognition, Automated - methods; Representations; Reproducibility of Results; Sensors; Skeleton; Wearable Electronic Devices; Wearable technology
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2020.3009448

For abnormal gait recognition, pattern-specific features indicating abnormalities are interleaved with the subject-specific differences representing biometric traits. Deep representations are, therefore, prone to overfitting, and the models derived cannot generalize well to new subjects. Furthermore, there is limited availability of abnormal gait data obtained from precise Motion Capture (Mocap) systems because of regulatory issues and slow adaptation of new technologies in health care. On the other hand, data captured from markerless vision sensors or wearable sensors can be obtained in home environments, but noises from such devices may prevent the effective extraction of relevant features. To address these challenges, we propose a cascade of deep architectures that can encode cross-modal and cross-subject transfer for abnormal gait recognition. Cross-modal transfer maps noisy data obtained from RGBD and wearable sensors to accurate 4-D representations of the lower limb and joints obtained from the Mocap system. Subsequently, cross-subject transfer allows disentangling subject-specific from abnormal pattern-specific gait features based on a multiencoder autoencoder architecture. To validate the proposed methodology, we obtained multimodal gait data based on a multicamera motion capture system along with synchronized recordings of electromyography (EMG) data and 4-D skeleton data extracted from a single RGBD camera. Classification accuracy was improved significantly in both Mocap and noisy modalities.