IterMask3D: Unsupervised anomaly detection and segmentation with test-time iterative mask refinement in 3D brain MRI

• Published in: Medical image analysis Vol. 107; no. Pt A; p. 103763
• Main Authors: Liang, Ziyun, Guo, Xiaoqing, Xu, Wentian, Ibrahim, Yasin, Voets, Natalie, Pretorius, Pieter M., Noble, J. Alison, Kamnitsas, Konstantinos
• Format: Journal Article
• Language: English
• Published: Netherlands Elsevier B.V 01.01.2026
• Subjects: 3D brain MRI; Algorithms; Anomaly detection; Brain - diagnostic imaging; Humans; Image Interpretation, Computer-Assisted - methods; Imaging, Three-Dimensional - methods; Magnetic Resonance Imaging - methods; Unsupervised anomaly segmentation; Unsupervised Machine Learning; 3D brain MRI; Anomaly detection; Unsupervised anomaly segmentation
• ISSN: 1361-8415; 1361-8423; 1361-8431; 1361-8423
• DOI: 10.1016/j.media.2025.103763

Unsupervised anomaly detection and segmentation methods train a model to learn the training distribution as ‘normal’. In the testing phase, they identify patterns that deviate from this normal distribution as ‘anomalies’. To learn the ‘normal’ distribution, prevailing methods corrupt the images and train a model to reconstruct them. During testing, the model attempts to reconstruct corrupted inputs based on the learned ‘normal’ distribution. Deviations from this distribution lead to high reconstruction errors, which indicate potential anomalies. However, corrupting an input image inevitably causes information loss even in normal regions, leading to suboptimal reconstruction and an increased risk of false positives. To alleviate this, we propose IterMask3D, an iterative spatial mask-refining strategy designed for 3D brain MRI. We iteratively spatially mask areas of the image as corruption and reconstruct them, then shrink the mask based on reconstruction error. This process iteratively unmasks ‘normal’ areas to the model, whose information further guides reconstruction of ‘normal’ patterns under the mask to be reconstructed accurately, reducing false positives. In addition, to achieve better reconstruction performance, we also propose using high-frequency image content as additional structural information to guide the reconstruction of the masked area. Extensive experiments on the detection of both synthetic and real-world imaging artifacts, as well as segmentation of various pathological lesions across multiple MRI sequences, consistently demonstrate the effectiveness of our proposed method. Code is available at https://github.com/ZiyunLiang/IterMask3D. [Display omitted] •Propose IterMask3D for 3D brain MRI anomaly segmentation and detection.•Reduce false positives via iterative spatial mask refinement during testing.•Guide reconstruction using high-frequency structural information.•Propose subject-specific thresholds to auto-stop mask refinement.•Evaluate on both artifact detection and pathology segmentation tasks.