Future of Computational Approaches for Understanding & Adapting User Interfaces

Towards Foundational UI Understanding

Computational UI understanding is a cornerstone for realizing intelligent user interfaces. The advance of deep models has enabled a wide spectrum of UI understanding tasks, ranging from UI captioning to grounding, which can potentially empower many interaction scenarios, such as accessibility and automation. In this talk, I will talk about several threads of UI understanding tasks conducted in my group, which illustrate our journey in exploring the space. These works pave the way for achieving foundational understanding of user interfaces. I discuss our recent work on Spotlight, a foundational UI model, where a single machine learning model can be tuned to address diverse UI tasks with substantial gains in accuracy. Finally, I will share my thoughts on the direction for developing foundational models for user interfaces.

Yang is a Senior Staff Research Scientist at Google Research, and an affiliate faculty member at University of Washington CSE. His research interests lie at the intersection of Human Computer Interaction and Artificial Intelligence, focusing on general deep learning research, and models for solving human interactive intelligence and improving user experiences. He pioneered on-device interactive ML on Android by developing impactful product features such as next app prediction and Gesture Search. Yang has extensively published in top venues across both the HCI and ML fields, including CHI, UIST, ICML, ACL, EMNLP, CVPR, NeurIPS (NIPS), ICLR and KDD, and has constantly served as area chairs or senior area (track) chairs across the fields. Yang is an editor of the Springer book on AI for HCI: A Modern Approach.

The Future of Mixed Reality Is Adaptive

Mixed Reality (MR) has the potential to transform the way we interact with digital information, and promises a rich set of applications, ranging from manufacturing and architecture to interaction with smart devices. Current MR approaches, however, are static, and users need to manually adjust the visibility, placement and appearance of their user interface every time they change their task or environment. This is distracting and leads to information overload. To overcome these challenges, we aim to understand and predict how users perceive and interact with digital information, and use this information in context-aware MR systems that automatically adapt when, where, and how to display virtual elements. We create computational approaches that leverage aspects such as users’ cognitive load or the semantic connection between the virtual elements and the surrounding physical objects. Our systems increase the applicability of MR, with the goal to seamlessly blend the virtual and physical world.

David Lindlbauer is an Assistant Professor at the Human-Computer Interaction Institute at Carnegie Mellon University, leading the Augmented Perception Lab. His research focusses on understanding how humans perceive and interact with digital information, and to build technology that goes beyond the flat displays of PCs and smartphones to advances our capabilities when interacting with the digital world. To achieve this, I create and study enabling technologies and computational approaches that control when, where and how virtual content is displayed to increase the usability of AR and VR interfaces.