Dynamic Crosswalk Scene Understanding for the Visually Impaired

• Published in: IEEE transactions on neural systems and rehabilitation engineering Vol. 29; pp. 1478 - 1486
• Main Authors: Tian, Shishun, Zheng, Minghuo, Zou, Wenbin, Li, Xia, Zhang, Lu
• Format: Journal Article
• Language: English
• Published: New York IEEE 2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE); IEEE Institute of Electrical and Electronics Engineers
• Subjects: Audio data; blind navigation; Cameras; crosswalk detection; Datasets; Electronic devices; Engineering Sciences; Global Positioning System; Image color analysis; Navigation; object detection; Object recognition; Pedestrian crossings; Pedestrian traffic flow; pedestrian traffic light recognition; Roads; Scene analysis; The visually impaired; Traffic; Traffic signals; Transforms; Vision sensors; Visual impairment; The visually impaired; blind navigation; Image color analysis; Navigation; Roads; pedestrian traffic light recognition; Transforms; Vision sensors; Cameras; object detection; crosswalk detection; Global Positioning System
• ISSN: 1534-4320; 1558-0210; 1558-0210
• DOI: 10.1109/TNSRE.2021.3096379

Independent mobility poses a great challenge to the visually impaired individuals. This paper proposes a novel system to understand dynamic crosswalk scenes, which detects the key objects, such as crosswalk, vehicle, and pedestrian, and identifies pedestrian traffic light status. The indication of where and when to cross the road is provided to the visually impaired based on the crosswalk scene understanding. Our proposed system is implemented on a head-mounted mobile device (SensingAI G1) equipped with an Intel RealSense camera and a cellphone, and provides surrounding scene information to visually impaired individuals through audio signal. To validate the performance of the proposed system, we propose a crosswalk scene understanding dataset which contains three sub-datasets: a pedestrian traffic light dataset with 7447 images, a dataset of key objects on the crossroad with 1006 images and a crosswalk dataset with 3336 images. Extensive experiments demonstrated that the proposed system was robust and outperformed the state-of-the-art approaches. The experiment conducted with the visually impaired subjects shows that the system is practical useful.