Multimodal Image-based Robotic Assistance & Guidance Environment (MIRAGE): A Novel Unified Framework for Real-Time Navigation and Visualization

MIRAGE, is a novel unified multimodal image-based Robotic Assistance and Guidance framework for real-time visualization and navigation for robotic surgeries. It integrates, stereo calibration, stereo 3D reconstruction (point clouds), AR video overlay, depth map, StereoSnap re-registration, TilePro-HUD navigation etc. on the da Vinci consoles, navigation/visualization, and fiducial-less ultrasound registration, with modules for,  tracker registration, and fiducial- rigid registration with live RMSE feedback. High-throughput video I/O and rendering run in C++, while a Python GUI interfaces via a zero-copy novel binding wrapper; stereo streaming and optical tracking operate in real time. MIRAGE was validated in phantom experiments across da Vinci S, Si, Xi, and SP platforms. Across clinically relevant working distances, mean 3D reprojection error was 0.6mm, with best 0.1mm RMSE at 100mm on Xi. Results support MIRAGE as a preclinical technical-validation platform for accurate, responsive, and extensible multimodal surgical guidance.