AI& AR in Surgical System
AI& AR in Surgical System
Embedded Files
Orion – Surgical AI Assistant integrates artificial intelligence and augmented reality to enhance robotic-assisted surgery. Through AI reasoning, voice recognition, and real-time AR visualization, Orion enables surgeons to access and interpret critical patient data hands-free during operations. The system combines Python (FastAPI), Unity, and WebSocket protocols to synchronize AI reasoning with AR overlays in real time. Designed and tested by three graduate students, Orion demonstrates how human-centred AI can improve surgical precision, safety, and decision-making.
Prototype Demo
Prototype Demo
Meet the Team
Meet the Team
Mohammed Adnan A
AR Integration & Unity Developer
Master of Digital Transformation and Innovation
Description: Built and optimized the Unity AR visualization layer. Managed the overlay display scripts, WebSocket integration, and event listener system to synchronize AI reasoning with real-time augmented reality feedback.
Constantin Koulong Mangeoue
AI & Backend Developer
Master of Electrical and Computer Engineering
Description: Designed and implemented the Intent Classifier module using Python (FastAPI) and OpenAI Whisper. Developed the logic for intent recognition, data retrieval, and recommendation features to enable real-time voice-command processing.
Nidhi Roy John
Testing & System Validation Lead Developer
Master of Electrical and Computer Engineering
Description: Led prototype testing and performance evaluation. Conducted latency, stability, and accuracy tests while coordinating user feedback sessions to refine system responsiveness and ensure reliability across all modules.
The Client and Technical Advisor
The Client and Technical Advisor
Client Name: Professor Ali Abbas
Associate Professor
The School of Engineering Design and Teaching Innovation, University of Ottawa
Role in Project: Served as client, providing conceptual guidance, design feedback, and direction on integrating AI reasoning into AR-assisted surgical workflows.
Technical Advisor: Professor Ismaeel
Adjunct Professor
Associate Professor of Computer Engineering, Kuwait College of Science and Technology
Role in Project： Offered technical mentorship on system architecture, performance optimization, and validation of real-time AI–AR interaction in the prototype environment.
Prorotype Demo
Prorotype Demo
Examples of Surgeon Prompts and LLM Responses.
Example of data retrieval
Intent classification subsystem.
JSON Response Example from Orion’s Recommendation Module.
Example of two intent types: Recommendation (1st picture) and
Data Retrieval (2nd Picture).
AR Display Output
AR Display Output.
Image Capture before and after segmentation.
From the Students
From the Students
Page updated
Google Sites
Report abuse