Prof. Dr. Philipp Fürnstahl
University of Zurich & Balgrist University Hospital

From Visualization to Understanding: Toward Context-Aware XR in Surgery.

Augmented Reality has long promised to transform surgery, and AR-based navigation solutions are by now clinically available. Yet their efficacy and efficiency in the operating room remain limited, and much of the technology's potential is still ahead of us. This keynote asks which aspects are crucial to move AR from a display technology toward a context-aware and intelligent system, spanning both treatment and training.

The first part revisits a decade of our translational research in orthopedics, spanning AR-based surgical navigation, instrument tracking, and approaches to knowledge and skill transfer. Reflecting on this work allowed us to identify the limitations and opportunities of AR in surgery, from hardware to visualization and user interfaces. These lessons have motivated a shift in our research: from delivering information toward understanding the surgical scene, the task, and the user.

The second part turns to our current research and the question of how we can understand the surgical context - a key enabler of the operating room of the future. Building toward physical AI, we position the Surgical Digital Twin as a central representation connecting sensing, perception, and understanding through Surgical Data Science. The Surgical Digital Twin integrates multimodal information, including visual, audio, and force data, to capture surgery holistically. Examples from our current research span surgical process understanding, activity recognition, and the integration of multimodal foundation models for spatial and semantic understanding of the surgical scene.

Looking ahead, understanding the surgical context will be central to making XR in surgery more effective and efficient, transforming it into a intelligent coordination layer that adapts to the procedure and orchestrates guidance, robotic execution, and workflow. This talk shares a vision for the future of XR in surgery, offered as an impulse for discussion.

Biography

Prof. Dr. Philipp Fürnstahl is Associate Professor at the University of Zurich and Head of the Research in Orthopedic Computer Science Lab, ROCS, at Balgrist University Hospital. He is also affiliated Professor of the ETH AI Center and the Digital Society Initiative. As a trained computer scientist with a PhD from ETH Zurich, his research spans Surgical Data Science and Computer- and Robot-Assisted Surgery, with a strong focus on clinical translation. He is co-founder of OR-X, a national research infrastructure for the surgery of the future, and Head of the 3D Planning and Printing Center at Balgrist. His work has led to several startups, patents, and medical devices, along with more than 200 peer-reviewed publications.

Web: Zurich | Balgrist  

Prof. Dr. Bruce Daniel
Stanford University

Challenges of looking inside the body with augmented reality.

Many medical interventions, including percutaneous needle biopsy, minimally invasive ablation, catheter and device placements, and surgical excisions are directed at invisible, non-palpable targets within opaque organs and tissues.  Augmented reality based on medical imaging has long promised the opportunity for physicians to directly look at virtual content in situ in the body to best plan and execute medical procedures with optima accuracy, including targeting of the entire abnormality while minimizing collateral damage to surrounding normal tissues.    But clinical utilization of true in situ augmented reality for targeting remains elusive.  One major barrier is that the human visual system is incorrectly perceiving the locations of targets below the surface.  In this talk we will discuss various depth cue errors when viewing internal targets at arms-length, and review published strategies to mitigate them, including transparency management, and virtual “holes” or apertures.  We will focus on the challenges of optical see-through AR systems, which are preferred for many general surgery applications.  Future research directions will also be discussed.

Biography

Dr. Daniel is Professor of Radiology and, by courtesy, of BioEngineering at Stanford University.  Having completed his medical degree at Harvard Medical School in 1990, he came to Stanford in 1995 as a National Cancer Institute cancer imaging research fellow.  He is a distinguished investigator of the Academy for Radiology and Biomedical Imaging Research and a fellow of the American Institute for Medical and Biological Imaging (AIMBE), and a fellow of the Society for Breast MRI.    Dr. Daniel’s research includes co-directing the IMMERS lab at Stanford which pursues mixed-reality solutions to many medical applications, including education, procedure planning and guidance based on medical imaging data.

Web: Stanford