Multimodal Few-Shot Learning for Gait Recognition

• Published in: Applied sciences Vol. 10; no. 21; p. 7619
• Main Authors: Moon, Jucheol, Le, Nhat Anh, Minaya, Nelson Hebert, Choi, Sang-Il
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.11.2020
• Subjects: Accelerometers; Accuracy; Algorithms; Biometrics; Datasets; Discriminant analysis; few-shot learning; Gait; gait analysis; multi-modality; Neural networks; open set recognition; Sensors; Smartphones; Support vector machines; Time series; wearable sensors
• ISSN: 2076-3417; 2076-3417
• DOI: 10.3390/app10217619

A person’s gait is a behavioral trait that is uniquely associated with each individual and can be used to recognize the person. As information about the human gait can be captured by wearable devices, a few studies have led to the proposal of methods to process gait information for identification purposes. Despite recent advances in gait recognition, an open set gait recognition problem presents challenges to current approaches. To address the open set gait recognition problem, a system should be able to deal with unseen subjects who have not included in the training dataset. In this paper, we propose a system that learns a mapping from a multimodal time series collected using insole to a latent (embedding vector) space to address the open set gait recognition problem. The distance between two embedding vectors in the latent space corresponds to the similarity between two multimodal time series. Using the characteristics of the human gait pattern, multimodal time series are sliced into unit steps. The system maps unit steps to embedding vectors using an ensemble consisting of a convolutional neural network and a recurrent neural network. To recognize each individual, the system learns a decision function using a one-class support vector machine from a few embedding vectors of the person in the latent space, then the system determines whether an unknown unit step is recognized as belonging to a known individual. Our experiments demonstrate that the proposed framework recognizes individuals with high accuracy regardless they have been registered or not. If we could have an environment in which all people would be wearing the insole, the framework would be used for user verification widely.