[GAZEploit]
Remote Keystroke Inference Attack by Gaze Estimation from Avatar Views in VR/MR Devices
Hanqiu Wang, Siqi Dai, Max Panoff, Shuo Wang from University of Florida, Haoqi Shan from CertiK Skyfall Team, Zihao Zhan from Texas Tech University
GAZEploit:
A novel attack that can infer eye-related biometrics from the avatar image to reconstruct text entered via gaze-controlled typing.
The GAZEploit attack leverages the vulnerability inherent in gaze-controlled text entry when users share a virtual avatar. Virtual avatars, whether shared through video calls, online meeting apps, live streaming platforms, or potentially malicious websites, pose a significant privacy risk by potentially exposing user information such as login credentials. By remotely capturing and analyzing the virtual avatar video, an attacker can reconstruct the typed keys. Notably, the GAZEploit attack is the first known attack in this domain that exploits leaked gaze information to remotely perform keystroke inference.
Demo Attack Video
A Youtube video of a demo on how GAZEploit infers the correct keystrokes from Persona view of Apple Vision pro when the victim is gaze typing.
Attack Overview
Our GAZEploit attack relies on two biometrics extracted from the Persona recordings, eye aspect ratio (EAR) and eye gaze estimation. We use supervised learning on these features to distinguish typing sessions from other VR-related activities such as watching videos or playing video games. Then, we map the gaze estimation directions on a virtual keyboard to determine the potential keystrokes.
Identifying Typing Session
The gaze estimation reveals a noteworthy pattern:
Direction of eye gaze tends to be more concentrated and exhibits a periodic pattern during typing session.
Frequency of eye blinking decreases during typing sessions
Recurrent neural network (RNN) is suitable for tasks that need to recognize patterns in sequential data.
We build an RNN with 128 hidden layers and use cross entropy as loss function. We use Adam Optimizer to update its weight during training.
We collect gaze-typing dataset from 30 participants and evaluate RNN performance. We can achieve a high accuracy, precision, and recall rate of 98.1%, 90.5%, and 97.2% when using 18 participants' data out of 30 to train and the rest to validate.
Identifying Individual Keystrokes
During gaze typing, users' gazes shift between keys and fixate on the key to be clicked, resulting in saccades followed by fixations. Saccades refers to the period when users move their gaze rapidly from one object to another. Fixations refers to the period when users stare at an object.
We developed an algorithm that calculate the stability of the gaze trace and set a threshold to classify fixations from saccades. We use the gaze estimation points in these high stability region as click candidates. Evaluation on our dataset shows precision and recall rate of 85.9% and 96.8% on identifying keystrokes within typing sessions.
Adaptive Virtual Keyboard Mapping
To accurately map gaze points to specific keys during individual keystrokes, it is essential to precisely determine the location of the virtual keyboard in virtual space. We can utilize eye-movement statistics to accurately estimate the keyboard's location.
The average normal vector of gaze direction indicate the plane where the virtual keyboard locates. Edge-located keystrokes such as 'Q', 'P', 'RETURN', 'NUMBERSPACE', 'SPACE' help determine the boundary of the virtual keyboard. Using these methods, we can determine both the location and the size of the virtual keyboard.
On the right figure we show how we map the fixation gaze directions to a virtual keyboard image. Our demo attack achieves a top-5 character prediction accuracy of 100%.
Evaluation
We conducted two rounds of data collection on 30 participants with different races, genders and ages. The evaluation proves GAZEploit effective on four typing scenarios including 1. message input, 2. password input, 3. Email address / URL input, and 4. passcode (PIN) input.
Top 1 Letter inference accuracy
The top 1 guess of keys on the QWERTY keyboard
Top 5 Letter inference accuracy
The top 5 guess of keys on the QWERTY keyboard
Top 1 Number inference accuracy
The top 1 guess of keys on the Number and special character keyboard
Top 5 Number inference accuracy
The top 5 guess of keys on the Number and special character keyboard
Top K character inference accuracy
Our Pre-print Paper is available at Arxiv.org.