A novel hybrid bidirectional unidirectional LSTM network for dynamic hand gesture recognition with Leap Motion

• Published in: Entertainment computing Vol. 35; p. 100373
• Main Authors: Ameur, Safa, Ben Khalifa, Anouar, Bouhlel, Med Salim
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.08.2020
• Subjects: Deep learning; Gaming; Hand gesture recognition; Leap Motion Controller; LSTM; Touchless interaction system; Deep learning; Touchless interaction system; LSTM; Leap Motion Controller; Gaming; Hand gesture recognition
• ISSN: 1875-9521; 1875-953X
• DOI: 10.1016/j.entcom.2020.100373

[Display omitted] •A detailed review on LSTM using all sensor modalities in action recognition field.•A first work with LMC that joins all LSTM variants with evaluation on two datasets.•An HBU-LSTM deep network based on the spatial and temporal dependencies. Due to the recent development of machine learning and sensor innovations, hand gesture recognition systems become promisingfor the digital entertainment field. In this paper, we propose a dynamic hand gesture recognition approach using touchless hand motions over a Leap Motion device. First, we analyze the sequential time series data gathered from Leap Motion using Long Short-Term Memory (LSTM) recurrent neural networks for recognition purposes. We exploit basic unidirectional LSTM and bidirectional LSTM separately. Then, we propound novel architecture by combining the aforementioned models with additional components to give a final prediction network, named Hybrid Bidirectional Unidirectional LSTM (HBU-LSTM). The suggested network improves the model performance significantly by considering the spatial and temporal dependencies between the Leap Motion data and the network layers during the forward and backward pass. The recognition models are examined on two available benchmark datasets, named the LeapGestureDB dataset and the RIT dataset. Experiments demonstrate the potential of the proposed HBU-LSTM network for dynamic hand gesture recognition, with an average recognition rate reaching approximately 90%. Our suggested approach reaches superior performance, in terms of accuracy and computational complexity, over some existing methods for hand gesture recognition.