Pose Estimation and Non-Rigid Registration for Augmented Reality During Neurosurgery

• Published in: IEEE transactions on biomedical engineering Vol. 69; no. 4; pp. 1310 - 1317
• Main Authors: Haouchine, Nazim, Juvekar, Parikshit, Nercessian, Michael, Wells, William, Golby, Alexandra, Frisken, Sarah
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.04.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE); Institute of Electrical and Electronics Engineers
• Subjects: Augmented Reality; Blood vessels; Brain; Brain - diagnostic imaging; Brain - surgery; Brain tumors; Computer Science; Datasets; Deformation; Feature extraction; image-guided intervention; Imaging, Three-Dimensional - methods; Magnetic Resonance Imaging; Neurosurgery; Pose estimation; Registration; Retrospective Studies; Surgery, Computer-Assisted - methods; Surgical mesh; Three-dimensional displays; Tumors; Visualization; Pose Estimation; Neurosurgery; Image-guided Intervention; Augmented Reality; Registration
• ISSN: 0018-9294; 1558-2531; 1558-2531
• DOI: 10.1109/TBME.2021.3113841

Objective: A craniotomy is the removal of a part of the skull to allow surgeons to have access to the brain and treat tumors. When accessing the brain, a tissue deformation occurs and can negatively influence the surgical procedure outcome. In this work, we present a novel Augmented Reality neurosurgical system to superimpose pre-operative 3D meshes derived from MRI onto a view of the brain surface acquired during surgery. Methods: Our method uses cortical vessels as main features to drive a rigid then non-rigid 3D/2D registration. We first use a feature extractor network to produce probability maps that are fed to a pose estimator network to infer the 6-DoF rigid pose. Then, to account for brain deformation, we add a non-rigid refinement step formulated as a Shape-from-Template problem using physics-based constraints that helps propagate the deformation to sub-cortical level and update tumor location. Results: We tested our method retrospectively on 6 clinical datasets and obtained low pose error, and showed using synthetic dataset that considerable brain shift compensation and low TRE can be achieved at cortical and sub-cortical levels. Conclusion: The results show that our solution achieved accuracy below the actual clinical errors demonstrating the feasibility of practical use of our system. Significance: This work shows that we can provide coherent Augmented Reality visualization of 3D cortical vessels observed through the craniotomy using a single camera view and that cortical vessels provide strong features for performing both rigid and non-rigid registration.