Abstract

Video-Assisted Thoracoscopic Surgery (VATS) heavily relies on the precise anatomical placement of surgical ports to navigate rigid instruments through narrow intercostal spaces. Currently, surgeons must cognitively map two-dimensional or three-dimensional preoperative medical imaging onto the patient's physical anatomy in the operating room—a highly subjective process prone to spatial error. While augmented reality (AR) provides a mechanism to project patient-specific three-dimensional digital twins directly onto the surgical field, achieving high-precision alignment historically requires cumbersome fiducial markers or expensive external optical tracking systems. This thesis presents the Human-In-the-Loop Mobile Augmented Reality for Thoracic Surgery (HILMARTS) framework, a novel, markerless spatial computing architecture deployed on consumer-grade, LiDAR-equipped mobile tablets. The core methodological contribution is the engineering of a two-stage, coarse-to-fine registration architecture. To bypass the local minima inherent in automated global alignment algorithms, the system leverages an interactive human-in-the-loop paradigm, allowing the surgical operator to perform an initial manual alignment. This coarse transformation is subsequently refined using an automated point-cloud registration pipeline combining RANSAC-based global initialization with Point-to-Plane Iterative Closest Point (ICP) refinement. To address the severe noise and specular artifacts with mobile Time-of-Flight sensors in clinical environments, the architecture integrates a rigorous segmentation-based data-cleaning pipeline. This incorporates deep learing-driven semantic segmentation via BlazePose for precise torso extraction and additional image processing steps for robust outlier rejection. Furthermore, this research introduces an occupancy-grid-based Surface Coverage Score, an algorithmic metric that provides real-time, actionable feedback on the geometric completeness of the captured point cloud to ensure the robustness of the ICP refinement. In a phantom study involving sixteen participants, the framework achieved a mean target registration error (mTRE) of 3.31±1.26 mm in approximately four minutes, falling within acceptable clinical accuracy standards for thoracic interventions. The system also demonstrated high usability, achieving a System Usability Scale (SUS) score of 84.1. This thesis demonstrates that accessible, markerless mobile AR has the potential to seamlessly integrate into surgical workflows, offering a viable and highly efficient alternative to bulky navigation systems.