DAGM-fusion: A dual-path CT-MRI image fusion model based multi-axial gated MLP

• Published in: Computers in biology and medicine Vol. 155; p. 106620
• Main Authors: Fan, Chao, Lin, Hao, Qiu, Yingying, Yang, Litao
• Format: Journal Article
• Language: English
• Published: United States Elsevier Ltd 01.03.2023; Elsevier Limited
• Subjects: Ablation; Algorithms; Bone density; Complexity; Computed tomography; Computer applications; Computer vision; Decision making; Decomposition; Deep learning; Design; Diagnostic systems; Dictionaries; Dual-path; Human tissues; Humans; Image fusion; Image Processing, Computer-Assisted - methods; Image quality; Information processing; Internal Medicine; Magnetic resonance imaging; Magnetic Resonance Imaging - methods; Medical diagnosis; Medical imaging; Modules; Multi-axial gated MLP; Neural networks; Other; Patch-loss; Pixels; Radiotherapy Planning, Computer-Assisted - methods; Soft tissues; Tomography; Tomography, X-Ray Computed - methods; Wavelet transforms; Dual-path; Image fusion; Multi-axial gated MLP; Patch-loss
• ISSN: 0010-4825; 1879-0534; 1879-0534
• DOI: 10.1016/j.compbiomed.2023.106620

Medical imaging technology provides a good understanding of human tissue structure. MRI provides high-resolution soft tissue information, and CT provides high-quality bone density information. By creating CT-MRI fusion images of complex diagnostic situations, experts can develop diagnoses and treatment plans more quickly and precisely. We propose a dual-path CT-MRI image fusion model based on multi-axial gated MLP to create high-quality CT-MRI fusion images. The model employs the feature fusion module SFT-block to effectively integrate detailed Local-Path information guided by global Global-Path information. The fusion is completed through triple constraints, namely global constraints, local constraints, and overall constraints. We design a multi-axial gated MLP module (Ag-MLP). The multi-axial structure maintains the computational complexity linear and increases MLP's inductive bias, allowing MLP to work in shallower or pixel-level small dataset tasks. Ag-MLP and CNN are combined in the network so that the model has both globality and locality. In addition, we design a loss calculation method based on image patches that adaptively generates weights for each patch based on image pixel intensity. The details of the image are efficiently increased when patch-loss is used. Numerous studies demonstrate that the results of our model are superior to those of the latest mainstream fusion model, which are more in accordance with actual clinical diagnostic standards. The ablation studies successfully validate the performance of the model's constituent parts. It is worth mentioning that the model can also be excellently generalized to other modal image fusion tasks. •We propose a CT-MRI fusion framework based on MLP with a dual-path and triple constraints.•A multi-axial gated MLP module (Ag-MLP) is presented.•We design an approach to calculating loss based on image patches that adaptively generates a weight for each patch.•The fusion images are more suitable for practical clinical diagnosis.•It can also handle other types of medical image fusion tasks quite well.