X2T: Training an X-to-Text Typing Interface with Online Learning from User Feedback
We aim to help users communicate their intent to machines using flexible, adaptive interfaces that translate arbitrary user input into desired actions. In this work, we focus on assistive typing applications in which a user cannot operate a keyboard, but can instead supply other inputs, such as webcam images that capture eye gaze or neural activity measured by a brain implant. Standard methods train a model on a fixed dataset of user inputs, then deploy a static interface that does not learn from its mistakes; in part, because extracting an error signal from user behavior can be challenging. We investigate a simple idea that would enable such interfaces to improve over time, with minimal additional effort from the user: online learning from user feedback on the accuracy of the interface's actions. In the typing domain, we leverage backspaces as feedback that the interface did not perform the desired action. We propose an algorithm called x-to-text (X2T) that trains a predictive model of the user's feedback, and uses this model to fine-tune any existing, default interface for translating user input into actions that select words or characters.
User input: eye gaze
We primarily evaluate X2T through a small-scale online user study with 12 participants who type sentences by gazing at their desired words.
The results show that X2T learns to outperform a non-adaptive default interface, and stimulates user co-adaptation to the interface.
O = X2T's gaze position estimate
O = default interface's gaze position estimate
User input: handwriting
We evaluate X2T through a large-scale observational study on handwriting samples from 60 users in the UJI Pen Characters dataset, to simulate gestural user input from a stylus or brain-computer interface.
The results show that X2T personalizes the interface to the unique handwriting style of individual users, and can leverage offline data collected from a default interface to improve its initial performance and accelerate online learning.
Action prediction accuracy in experiments that train on data from user i and evaluate on data from user j
User input: signals from brain implant
The results show that X2T adapts to recent input-action-reward data and outperforms a default interface that is trained via supervised learning on older, paired input-action data.