Online Handwriting Recognition using LSTM on Microcontroller and IMU Sensors

• Published in: 2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA) pp. 999 - 1004
• Main Authors: Meisl, Florian, Eibensteiner, Florian, Petz, Phillip, Langer, Josef
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.12.2022
• Subjects: Edge Computing; Handwriting recognition; IMU Sensor; LSTM; Machine Learning; Microcontroller; Microcontrollers; Online Handwriting Recognition; Prototypes; Quantization (signal); Real-time systems; Symbols; Training
• DOI: 10.1109/ICMLA55696.2022.00167

The trend toward the Internet of Things has led to a rapid increase in the amount of data that needs to be processed. Artificial intelligence (AI) can serve as a very helpful tool to extract or compress essential information of data. However, AI places high demands on a system's hardware. This is not exactly in line with the strengths of embedded systems.This paper combines AI on embedded systems with the not-yet fully explored subject of online handwriting recognition (HWR). The main contribution is the deployment and real-time operation of AI on a microcontroller (MCU). Model architectures using long short-term memory (LSTM) cells and 1D convolutional neural networks (CNNs) are used to process live data from inertial measurement units (IMUs) sensors. The dataset used for training the AI models was recorded with a self-developed prototype. After training, the models are converted and deployed on a MCU. The conversion process includes quantization from a 32-bit floating-point to an 8-bit fixed-point datatype. The TensorFlow Lite Micro (TFLM) framework is used to run inference on the MCU. For predictions in real-time optimizations are applied to the framework, which results in running inference approx. 827 times faster. The optimized AI model implementation is then used to classify handwritten characters using the live data from the IMU sensors. This first approach has shown, that the separation of the symbols is necessary to be able to classify characters from live sensor data with high accuracy.