From GOMS to Deep Reinforcement Learning
CHI 2023 Onsite Course, April 26
Learn to understand, create, and utilize computational cognitive models in HCI contexts.
Use Python notebooks to easily simulate interactive task behaviour in various environments.
Topics include computational rationality, deep reinforcement learning, parameter inference, model design
This course introduces computational cognitive modeling for researchers and practitioners in the field of HCI. Cognitive models use computer programs to model how users perceive, think, and act in human--computer interaction. They offer a powerful approach for understanding interactive tasks and improving user interfaces. This course starts with a review of classic architecture based models such as GOMS and ACT-R. It then rapidly progresses to introducing modern modelling approaches powered by machine learning methods, in particular deep learning, reinforcement learning (RL), and deep RL. The course is built around hands-on Python programming using notebooks.
The course consists of four 90 minute modules. All modules make use of Python Notebooks, which are found at the public repository https://github.com/howesa/CHI22-CogMod-Tutorial (please see the repository documentation for installation instructions).
Introduction to cognitive modeling. In this session we will critically review the history of cognitive modeling in HCI over the past 50 years. The review will cover modeling approaches that include mathematical models of movement time, procedural models of skill, models of eye movements, computational simulations of human information processing, models inspired by how animals forage for food and models of human learning, rationality and decision making. After completing the section the attendees will understand the role of computational cognitive modeling in HCI, and know how the historical approaches have converged into the state-of-the-art of modeling.
Deep learning. The session introduces deep learning as an approach to building cognitive models in HCI. Deep learning has been successfully used in modelling various types of interactive behaviour, such as visual search and mid-air interaction in VR. One of the advantages of deep learning is in its ability to discover features and structure in data that are important for explanation and prediction. Deep learning models also excel at processing input data that have a large feature space and contain vast amounts of information. After completing this section, attendees understand the role of deep learning in HCI models, and have experience in applying such models to predict user behavior with realistic UIs.
Reinforcement learning. The session provides an introduction to creating RL based simulation models of interactive tasks. The module covers basic tabular methods (temporal difference RL) and advanced, deep RL approaches, with the focus on how these techniques have been used in recent breaktrough models in HCI and machine learning. We cover the theoretical underpinnings of RL as the basis of simulating human adaptation in interactive tasks. The session covers modeling of such phenomena as multitasking, typing, and visual search. After completing this section, attendees will be able to read and understand research papers in HCI that make use of RL based computational cognitive models, make us of such models in their work, and participate in designing them further.
Building a model. How to construct a model for a practical purpose? In this unit, we adapt the modern modeling workflow to the case of creating a computational rational model in HCI.
Schedule and Registration
The course fills one conference day (from early morning to late afternoon), and is only available on-site at Hamburg.
April 26. 9:00-18:00
Introduction to cognitive modeling
Creating a model