MAEST: accurately spatial domain detection in spatial transcriptomics with graph masked autoencoder

• Published in: Briefings in bioinformatics Vol. 26; no. 2
• Main Authors: Zhu, Pengfei, Shu, Han, Wang, Yongtian, Wang, Xiaofeng, Zhao, Yuan, Hu, Jialu, Peng, Jiajie, Shang, Xuequn, Tian, Zhen, Chen, Jing, Wang, Tao
• Format: Journal Article
• Language: English
• Published: England Oxford Publishing Limited (England) 04.03.2025
• Subjects: Accuracy; Algorithms; Animals; Autoencoder; Brain; Brain - metabolism; Clustering; Computational Biology - methods; Gene expression; Gene Expression Profiling - methods; Graph neural networks; Graphical representations; Hippocampus - metabolism; Humans; Mice; Neural networks; Neural Networks, Computer; Olfactory bulb; Software; Source code; Spatial data; Transcriptome; Transcriptomics; graph masked autoencoder; spatial domain identification; graph contrastive learning; spatial transcriptomics; joint domain detection
• ISSN: 1467-5463; 1477-4054; 1477-4054
• DOI: 10.1093/bib/bbaf086

Spatial transcriptomics (ST) technology provides gene expression profiles with spatial context, offering critical insights into cellular interactions and tissue architecture. A core task in ST is spatial domain identification, which involves detecting coherent regions with similar spatial expression patterns. However, existing methods often fail to fully exploit spatial information, leading to limited representational capacity and suboptimal clustering accuracy. Here, we introduce MAEST, a novel graph neural network model designed to address these limitations in ST data. MAEST leverages graph masked autoencoders to denoise and refine representations while incorporating graph contrastive learning to prevent feature collapse and enhance model robustness. By integrating one-hop and multi-hop representations, MAEST effectively captures both local and global spatial relationships, improving clustering precision. Extensive experiments across diverse datasets, including the human brain, mouse hippocampus, olfactory bulb, brain, and embryo, demonstrate that MAEST outperforms seven state-of-the-art methods in spatial domain identification. Furthermore, MAEST showcases its ability to integrate multi-slice data, identifying joint domains across horizontal tissue sections with high accuracy. These results highlight MAEST’s versatility and effectiveness in unraveling the spatial organization of complex tissues. The source code of MAEST can be obtained at https://github.com/clearlove2333/MAEST.