Virtual Keymysteries Unveiled (Safe Things '24)

Given the growing popularity of VR headsets, concerns about security vulnerabilities also increase. With headsets like Meta Quest and now the Apple Vision Pro, users are generally required to sign-in to gain full access to all of the headset's features, whether that's linking other devices or downloading applications/games. As such, a sign-in and password key must be typed using a virtual keyboard. In the case of the Meta Quest, users are allowed the option of hand tracking and using the index finger to type letters, or using the controllers to aim a virtual beam at the desired letters.

Despite this seemingly harmless setup, it is possible that malicious attackers can exploit this typing through simply acquiring a clear video stream of a user virtually typing their credentials. Given this video stream and knowing the layout of the QWERTY keyboard, they could create a keylogger to extract private information. Such an attack was demonstrated by this paper and showed that among 500 attacks, they had a range of 40 to 87% accuracy. Although this is a large variance, it still shows that the attack is possible with further refinements. 

As such, we proposed a method of circumventing this attack by varying the key positions and sizes during each keystroke such that ordinary key placements would consistently overlap each other while maintaining the QWERTY layout. Although this is a relatively simple method, it would theoretically result in enough confusion for a keylogging inference attack to be mitigated. 

I personally implemented the entire keyboard and randomization method shown below in the demo. 

Conference: https://safe-things-2024.github.io/

Publication: Link Pending

Acknowledgements:

This project was my 3rd successful publication and I give thanks to my colleagues, as well as my advisor Professor Nader Sehatbakhsh.