Formal Research

Using Google Cardboard, I made the first AR headset experience that does not rely on expensive specialized hardware. By reconstructing and rendering the user's surroundings in VR, the AR MultiPendulum app replicated Microsoft Hololens. The process of creating the app involved heavy optimization for power efficiency and fixing bugs caused by using a GPU for general-purpose tasks. The final product was ARHeadsetKit, a high-level open-source framework for creating AR headset experiences.

In October 2020, Apple released the iPhone 12 Pro, which had a LiDAR scanner that enabled it to reconstruct the 3D shape of its surrounding scene. Coupling the Google Cardboard VR viewer with scene reconstruction allows the real world to be rendered in VR, replicating the experience of using an AR headset such as Hololens. In order to enhance this AR headset experience, I created software that mapped color in a video stream to areas on a reconstructed scene mesh, and retained that color when the mesh expanded and changed shape as the LiDAR scanner gathered new data. By adding color to scene reconstruction, the experience became more realistic, allowing the user to view the color of their surroundings in their peripheral vision and in areas occluded from the camera's view. This software runs on an iPhone, providing users with an accessible way to experience an AR headset.