Multi-modal imaging, model-based tracking, and mixed reality visualisation for orthopaedic surgery

• Published in: Healthcare technology letters Vol. 4; no. 5; pp. 168 - 173
• Main Authors: Lee, Sing Chun, Fuerst, Bernhard, Tateno, Keisuke, Johnson, Alex, Fotouhi, Javad, Osgood, Greg, Tombari, Federico, Navab, Nassir
• Format: Journal Article
• Language: English
• Published: England The Institution of Engineering and Technology 01.10.2017; John Wiley & Sons, Inc; Wiley
• Subjects: 3D point clouds; adapted 3D model-based tracking algorithm; Algorithms; automatic tool segmentation; bone; Calibration; Cameras; CBCT imaging; complex 3D structures; computerised tomography; cone-beam computed tomography imaging space; diagnostic radiography; entry point; image fusion; image reconstruction; image registration; image segmentation; interactive 3D mixed reality environment; Intervention; iterative closest point algorithm; iterative methods; Localization; medical image processing; mixed reality environment; mixed reality visualisation; mobile C-arm; model-based surgical tool tracking; multimodal data fusion; multimodal imaging; Navigation systems; Optics; orthopaedic surgery; orthopaedics; Orthopedics; partial occlusion; pelvis; Radiation; real-time automatic fusion; reconstructed surface; red-green-blue-depth camera; screw placement; Special Issue on Augmented Environments for Computer-Assisted Interventions; Surgeons; surgery; Surgical apparatus & instruments; surgical tools; synthetic fluoroscopic images; target registration error; tracking accuracy; two-dimensional fluoroscopic images; workflow; iterative methods; orthopaedics; tracking accuracy; image fusion; image registration; red-green-blue-depth camera; pelvis; automatic tool segmentation; surgical tools; model-based surgical tool tracking; cone-beam computed tomography imaging space; partial occlusion; synthetic fluoroscopic images; image segmentation; interactive 3D mixed reality environment; mixed reality environment; reconstructed surface; adapted 3D model-based tracking algorithm; complex 3D structures; medical image processing; multimodal data fusion; two-dimensional fluoroscopic images; target registration error; orthopaedic surgery; workflow; mobile C-arm; iterative closest point algorithm; diagnostic radiography; bone; image reconstruction; real-time automatic fusion; multimodal imaging; mixed reality visualisation; computerised tomography; 3D point clouds; CBCT imaging; screw placement; entry point; surgery
• ISSN: 2053-3713; 2053-3713
• DOI: 10.1049/htl.2017.0066

Orthopaedic surgeons are still following the decades old workflow of using dozens of two-dimensional fluoroscopic images to drill through complex 3D structures, e.g. pelvis. This Letter presents a mixed reality support system, which incorporates multi-modal data fusion and model-based surgical tool tracking for creating a mixed reality environment supporting screw placement in orthopaedic surgery. A red–green–blue–depth camera is rigidly attached to a mobile C-arm and is calibrated to the cone-beam computed tomography (CBCT) imaging space via iterative closest point algorithm. This allows real-time automatic fusion of reconstructed surface and/or 3D point clouds and synthetic fluoroscopic images obtained through CBCT imaging. An adapted 3D model-based tracking algorithm with automatic tool segmentation allows for tracking of the surgical tools occluded by hand. This proposed interactive 3D mixed reality environment provides an intuitive understanding of the surgical site and supports surgeons in quickly localising the entry point and orienting the surgical tool during screw placement. The authors validate the augmentation by measuring target registration error and also evaluate the tracking accuracy in the presence of partial occlusion.