Applying Hybrid Deep Neural Network for the Recognition of Sign Language Words Used by the Deaf COVID-19 Patients

• Published in: Arabian journal for science and engineering Vol. 48; no. 2; pp. 1349 - 1362
• Main Authors: Venugopalan, Adithya, Reghunadhan, Rajesh
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.02.2023; Springer Nature B.V
• Subjects: Artificial neural networks; Computer Engineering and Computer Science; Coronaviruses; COVID-19; Datasets; Engineering; Health care; Humanities and Social Sciences; multidisciplinary; Norms; Patients; Recognition; Research Article-Computer Engineering and Computer Science; Science; Sign language; Viral diseases; Viruses; COVID-19; Deep learning; Indian sign language; Emergency words; Hand gesture recognition
• ISSN: 2193-567X; 1319-8025; 2191-4281; 2191-4281
• DOI: 10.1007/s13369-022-06843-0

The rapid spread of the novel corona virus disease (COVID-19) has disrupted the traditional clinical services all over the world. Hospitals and healthcare centers have taken extreme care to minimize the risk of exposure to the virus by restricting the visitors and relatives of the patients. The dramatic changes happened in the healthcare norms have made it hard for the deaf patients to communicate and receive appropriate care. This paper reports a work on automatic sign language recognition that can mitigate the communication barrier between the deaf patients and the healthcare workers in India. Since hand gestures are the most expressive components of a sign language vocabulary, a novel dataset of dynamic hand gestures for the Indian sign language (ISL) words commonly used for emergency communication by deaf COVID-19 positive patients is proposed. A hybrid model of deep convolutional long short-term memory network has been utilized for the recognition of the proposed hand gestures and achieved an average accuracy of 83.36%. The model performance has been further validated on an alternative ISL dataset as well as a benchmarking hand gesture dataset and obtained average accuracies of 97 % and 99.34 ± 0.66 % , respectively.