Integrating Eye Tracking With Grouped Fusion Networks for Semantic Segmentation on Mammogram Images

• Published in: IEEE transactions on medical imaging Vol. 44; no. 2; pp. 868 - 879
• Main Authors: Xie, Jiaming, Zhang, Qing, Cui, Zhiming, Ma, Chong, Zhou, Yan, Wang, Wenping, Shen, Dinggang
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.02.2025
• Subjects: Algorithms; Annotations; Biomedical imaging; Breast - diagnostic imaging; Breast Neoplasms - diagnostic imaging; Computer aided diagnosis; Computer vision; Deep Learning; Diseases; eye tracking; Eye-Tracking Technology; Female; Gaze tracking; Humans; Image Processing, Computer-Assisted - methods; mammogram image; Mammography - methods; Medical diagnostic imaging; medical image segmentation; Neural Networks, Computer; Semantic segmentation; Semantics; Solid modeling; Transformers; Visualization
• ISSN: 0278-0062; 1558-254X; 1558-254X
• DOI: 10.1109/TMI.2024.3468404

Medical image segmentation has seen great progress in recent years, largely due to the development of deep neural networks. However, unlike in computer vision, high-quality clinical data is relatively scarce, and the annotation process is often a burden for clinicians. As a result, the scarcity of medical data limits the performance of existing medical image segmentation models. In this paper, we propose a novel framework that integrates eye tracking information from experienced radiologists during the screening process to improve the performance of deep neural networks with limited data. Our approach, a grouped hierarchical network, guides the network to learn from its faults by using gaze information as weak supervision. We demonstrate the effectiveness of our framework on mammogram images, particularly for handling segmentation classes with large scale differences. We evaluate the impact of gaze information on medical image segmentation tasks and show that our method achieves better segmentation performance compared to state-of-the-art models. A robustness study is conducted to investigate the influence of distraction or inaccuracies in gaze collection. We also develop a convenient system for collecting gaze data without interrupting the normal clinical workflow. Our work offers novel insights into the potential benefits of integrating gaze information into medical image segmentation tasks.