VR Object Detection & Blocker
VR Object Detection & Blocker
(w/ Alexa Hatcher | Mentored by: Adam Kohl)
Jun 2025: ~20hrs
Embedded Files
The goal:
The goal:
Broadly: Learn about how to integrate existing machine learning models with Unity and VR.
Specifically: Implement object detection for VR that runs on a desktop (not on the headset).
What I learned:
What I learned:
Gained general proficiency in Unity
TCP and UDP connections between Unity and Python
VR headsets do not give you access to their raw camera feeds on desktop (for Quest, you have to be built to the headset to have access)
Developing for research requires consistency, which consumer hardware is not designed for
Bounding Boxes
Bounding Boxes
Block Out Specified Objects
Block Out Specified Objects
Webcam Alignment Visualization
Webcam Alignment Visualization
Software Design:
Software Design:
Get images from webcam and send to Unity
Unity sends image as byte string over TCP connection to Python server
Python uses YOLO to get data about scene
Unity receives bounding box information and draws boxes aligned with headset view
Final output is rendered to the headset
Challenges:
Webcam had to be used since headset cameras could not be used directly
Aligning webcam view with headset view is very difficult. This project used ad hoc fix, but homography would be interesting to implement to get a better alignment
Page updated
Google Sites
Report abuse