GeSeNet: A General Semantic-Guided Network With Couple Mask Ensemble for Medical Image Fusion

• Published in: IEEE transaction on neural networks and learning systems Vol. 35; no. 11; pp. 16248 - 16261
• Main Authors: Li, Jiawei, Liu, Jinyuan, Zhou, Shihua, Zhang, Qiang, Kasabov, Nikola K.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.11.2024
• Subjects: Algorithms; Deep Learning; Discrete wavelet transforms; Feature extraction; Humans; Image edge detection; Image fusion; Image Processing, Computer-Assisted - methods; Magnetic resonance imaging; Magnetic Resonance Imaging - methods; Medical diagnostic imaging; Multimodal Imaging - methods; multimodal medical image; Neural Networks, Computer; Positron-Emission Tomography - methods; region mask; semantic information; Semantics; Tomography, Emission-Computed, Single-Photon - methods; Tomography, X-Ray Computed - methods
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2023.3293274

At present, multimodal medical image fusion technology has become an essential means for researchers and doctors to predict diseases and study pathology. Nevertheless, how to reserve more unique features from different modal source images on the premise of ensuring time efficiency is a tricky problem. To handle this issue, we propose a flexible semantic-guided architecture with a mask-optimized framework in an end-to-end manner, termed as GeSeNet. Specifically, a region mask module is devised to deepen the learning of important information while pruning redundant computation for reducing the runtime. An edge enhancement module and a global refinement module are presented to modify the extracted features for boosting the edge textures and adjusting overall visual performance. In addition, we introduce a semantic module that is cascaded with the proposed fusion network to deliver semantic information into our generated results. Sufficient qualitative and quantitative comparative experiments (i.e., MRI-CT, MRI-PET, and MRI-SPECT) are deployed between our proposed method and ten state-of-the-art methods, which shows our generated images lead the way. Moreover, we also conduct operational efficiency comparisons and ablation experiments to prove that our proposed method can perform excellently in the field of multimodal medical image fusion. The code is available at https://github.com/lok-18/GeSeNet .