IoT and AI-Based Application for Automatic Interpretation of the Affective State of Children Diagnosed with Autism

• Published in: Sensors (Basel, Switzerland) Vol. 22; no. 7; p. 2528
• Main Authors: Popescu, Aura-Loredana, Popescu, Nirvana, Dobre, Ciprian, Apostol, Elena-Simona, Popescu, Decebal
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 25.03.2022; MDPI
• Subjects: Algorithms; Autism; autism spectrum disorder; Autistic children; Autistic Disorder - diagnosis; Bluetooth technology; Child; Classrooms; Data mining; drawing interpretation; Emotions; Feedback; Health aspects; humanoid robots; Humans; Machine Learning; Mobile communications networks; Musical performances; Neural networks; Neural Networks, Computer; Parents & parenting; Robotics; Robots; Sign language; Social skills; Tutoring; Wireless telephone software; South Korea; humanoid robots; drawing interpretation; autism spectrum disorder; neural networks
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s22072528

In the context in which it was demonstrated that humanoid robots are efficient in helping children diagnosed with autism in exploring their affective state, this paper underlines and proves the efficiency of a previously developed machine learning-based mobile application called PandaSays, which was improved and integrated with an Alpha 1 Pro robot, and discusses performance evaluations using deep convolutional neural networks and residual neural networks. The model trained with MobileNet convolutional neural network had an accuracy of 56.25%, performing better than ResNet50 and VGG16. A strategy for commanding the Alpha 1 Pro robot without its native application was also established and a robot module was developed that includes the communication protocols with the application PandaSays. The output of the machine learning algorithm involved in PandaSays is sent to the humanoid robot to execute some actions as singing, dancing, and so on. Alpha 1 Pro has its own programming language—Blockly—and, in order to give the robot specific commands, Bluetooth programming is used, with the help of a Raspberry Pi. Therefore, the robot motions can be controlled based on the corresponding protocols. The tests have proved the robustness of the whole solution.