UAE: Universal Anatomical Embedding on multi-modality medical images

• Published in: Medical image analysis Vol. 103; p. 103562
• Main Authors: Bai, Xiaoyu, Bai, Fan, Huo, Xiaofei, Ge, Jia, Lu, Jingjing, Ye, Xianghua, Shu, Minglei, Yan, Ke, Xia, Yong
• Format: Journal Article
• Language: English
• Published: Netherlands Elsevier B.V 01.07.2025
• Subjects: Algorithms; Anatomic Landmarks - diagnostic imaging; Anatomical embedding learning; Humans; Image Interpretation, Computer-Assisted - methods; Landmark matching; Magnetic Resonance Imaging - methods; Multi-modality image alignment; Multimodal Imaging - methods; Tomography, X-Ray Computed - methods; Landmark matching; Anatomical embedding learning; Multi-modality image alignment
• ISSN: 1361-8415; 1361-8423; 1361-8423
• DOI: 10.1016/j.media.2025.103562

Identifying anatomical structures (e.g., lesions or landmarks) is crucial for medical image analysis. Exemplar-based landmark detection methods are gaining attention as they allow the detection of arbitrary points during inference without needing annotated landmarks during training. These methods use self-supervised learning to create a discriminative voxel embedding and match corresponding landmarks via nearest-neighbor searches, showing promising results. However, current methods still face challenges in (1) differentiating voxels with similar appearance but different semantic meanings (e.g., two adjacent structures without clear borders); (2) matching voxels with similar semantics but markedly different appearance (e.g., the same vessel before and after contrast injection); and (3) cross-modality matching (e.g., CT-MRI landmark-based registration). To overcome these challenges, we propose a Unified framework for learning Anatomical Embeddings (UAE). UAE is designed to learn appearance, semantic, and cross-modality anatomical embeddings. Specifically, UAE incorporates three key innovations: (1) semantic embedding learning with prototypical contrastive loss; (2) a fixed-point-based matching strategy; and (3) an iterative approach for cross-modality embedding learning. We thoroughly evaluated UAE across intra- and inter-modality tasks, including one-shot landmark detection, lesion tracking on longitudinal CT scans, and CT-MRI affine/rigid registration with varying fields of view. Our results suggest that UAE outperforms state-of-the-art methods, offering a robust and versatile approach for landmark-based medical image analysis tasks. Code and trained models are available at: https://github.com/alibaba-damo-academy/self-supervised-anatomical-embedding-v2. •We proposed a unified framework for learning semantic and appearance embeddings.•We introduced a fixed-point-based robust landmark matching technique.•We proposed an iterative method that enables multi-modality embedding learning.