The integration of Extended Reality (XR) into education is revolutionising on-site, remote, and hybrid learning environments. However, delivering truly intelligent XR experiences requires robust AI models operating in parallel. A key application involves using computer vision to observe and reason about student motions and behaviours, as well as other common objects present in a hybrid teaching space.

In this specific use case, we are providing an XR experience for a medical laboratory, utilising AI-powered cameras to autonomously capture photos and videos for remote participants. Training these machine vision algorithms demands massive, diverse datasets. However, capturing this physical motion data across countless real-world scenarios is complex, expensive, and an extremely time-consuming process.

This four-week winter scholarship addresses the data bottleneck by developing a digital twin prototype. The project uses the NVIDIA Omniverse Kit (specifically NVIDIA USD Composer and Replicator) to explore the generation of high-fidelity synthetic datasets for training computer vision models. By simulating the physical environment and applying domain randomisation to student motions and behaviours, we can rapidly generate vast amounts of diverse, fully annotated synthetic data. This approach bypasses the limitations of real-world data collection, providing a scalable, cost-effective pipeline to train advanced computer vision models for intelligent educational systems. 

Important: This project only focuses specifically on the digital twin (simulated medical laboratory), and will not involve any AI development or AI training. NO background in AI is required.

This project contributes to several research initiatives within Monash’s Embodied Visualisation (EmVis) Group. EmVis recently introduced a new theme, ‘IAxAI’, which integrates AI models into immersive environments. As many projects targeting specific use cases are constrained by limited datasets, this work will benefit a wide range of applications, and the skills gained will facilitate future contributions and collaborations.

Please ensure you possess the required skills listed below, then apply through Winter Scholarships application.

Required Skills:
General software engineering and programming knowledge

Fundamental 3D graphics knowledge

Animation development knowledge