Multimodal Image-based Robotic Assistance & Guidance Environment (MIRAGE): A Novel Unified Framework for Real-Time Navigation and Visualization

The MIRAGE, is a novel unified multimodal image-based Robotic Assistance and Guidance framework for real-time visualization and navigation for robotic surgeries. It integrates stereo 3D reconstruction (point clouds), AR video overlay, TilePro HUD on the da Vinci console, navigation/visualization, and ultrasound, with modules for stereo calibration, tracker registration, and fiducial-based rigid registration with live RMSE feedback. High-throughput video I/O and rendering run in C++, while a Python GUI interfaces via a zero-copy novel binding wrapper; stereo streaming and optical tracking operate in real time. Across 18 experiments on da Vinci S, Si, Xi, and SP systems, MIRAGE auto-adapts to platform-specific endoscopes and video formats. Stereo inference averaged ~0.20 s per pair. Over clinically relevant working distances, mean 3D reprojection error was ~0.6 mm, with best 0.1 mm RMSE at 100 mm on Xi. Results show responsive, accurate, extensible multimodal guidance for research and translational workflows. 

A U.S. provisional patent application covering aspects of this work has been filed (U.S. provisional application No. 63/920,165).