Multi-Modal Instrument Recognition Method Based on Improved YOLOv5s and ESPNet

• Published in: IEEE International Conference on Power, Intelligent Computing and Systems (Online) pp. 549 - 555
• Main Authors: Shen, Shixian, Wang, Jianguo, Feng, Yong, Li, Yingna
• Format: Conference Proceeding
• Language: English
• Published: IEEE 26.07.2024
• Subjects: Accuracy; Analytical models; Anomaly detection; Data models; digital twin; dynamic non-monotonic focus mechanism; ESPNet; Focusing; Image segmentation; Industries; Instruments; Monitoring; pointer-type instrument; YOLO
• ISSN: 2834-8567
• DOI: 10.1109/ICPICS62053.2024.10797062

Addressing the challenges of high cost, low accuracy, and poor real-time performance in anomaly detection within industrial production processes, this study proposes a multi-modal instrument recognition method based on improved YOLOv5s and ESPNet. Specifically, a dynamic non-monotonic focusing mechanism is introduced into the YOLOv5s object detection model, enhancing its ability to accurately detect the position of pointer-type instruments. Simultaneously, affine and perspective transformations are employed to rectify skewed and rotated images. Secondly, variable convolutions are integrated into the ESPNet segmentation network to adaptively capture nonlinear deformation elements of pointers and dials in images, thereby extracting comprehensive key information from both. Subsequently, a Hough transform operation is applied to fit the straight line representing the pointer, while contour tracking methods are utilized to extract the valid range of dial data. Based on linear proportional relationships, the reading of pointer-type instruments is computed. Finally, a comprehensive model is built by integrating image-acquired instrument data with corresponding sensor data, enabling anomaly detection from a multimodal data analysis perspective. Experimental results demonstrate an identification accuracy of 96.21% for pointer-type instruments, with an average detection speed of 0.216 seconds and a data anomaly detection rate of 99.43%. This method accurately and rapidly identifies data anomalies, meeting the demands for key indicator monitoring in industrial production.