Cognitive modeling

From GOMS to Deep Reinforcement Learning

CHI 2022 Online Course, April 26-27

Course highlights

  • Learn to understand, create, and utilize computational cognitive models in HCI contexts.

  • Use Python notebooks to easily simulate interactive task behaviour in various environments.

  • Topics include computational rationality, deep reinforcement learning, parameter inference, model design

This course introduces computational cognitive modeling for researchers and practitioners in the field of HCI. Cognitive models use computer programs to model how users perceive, think, and act in human--computer interaction. They offer a powerful approach for understanding interactive tasks and improving user interfaces. This course starts with a review of classic architecture based models such as GOMS and ACT-R. It then rapidly progresses to introducing modern modelling approaches powered by machine learning methods, in particular deep learning, reinforcement learning (RL), and deep RL. The course is built around hands-on Python programming using notebooks.

Course contents

The course consists of four 90 minute modules. All modules make use of Python Notebooks, which are found at the public repository https://github.com/howesa/CHI22-CogMod-Tutorial (please see the repository documentation for installation instructions).

  • Introduction to cognitive modeling. In this session we will critically review the history of cognitive modeling in HCI over the past 50 years. The review will cover modeling approaches that include mathematical models of movement time, procedural models of skill, models of eye movements, computational simulations of human information processing, models inspired by how animals forage for food and models of human learning, rationality and decision making. After completing the section the attendees will understand the role of computational cognitive modeling in HCI, and know how the historical approaches have converged into the state-of-the-art of modeling.

  • Deep learning. The session introduces deep learning as an approach to building cognitive models in HCI. Deep learning has been successfully used in modelling various types of interactive behaviour, such as visual search and mid-air interaction in VR. One of the advantages of deep learning is in its ability to discover features and structure in data that are important for explanation and prediction. Deep learning models also excel at processing input data that have a large feature space and contain vast amounts of information. After completing this section, attendees understand the role of deep learning in HCI models, and have experience in applying such models to predict user behavior with realistic UIs.

  • Reinforcement learning. The session provides an introduction to creating RL based simulation models of interactive tasks. The module covers basic tabular methods (temporal difference RL) and advanced, deep RL approaches, with the focus on how these techniques have been used in recent breaktrough models in HCI and machine learning. We cover the theoretical underpinnings of RL as the basis of simulating human adaptation in interactive tasks. The session covers modeling of such phenomena as multitasking, typing, and visual search. After completing this section, attendees will be able to read and understand research papers in HCI that make use of RL based computational cognitive models, make us of such models in their work, and participate in designing them further.

  • Modeling workflow. How to construct a model for a practical purpose? In this unit, we adapt the modern modeling workflow [5] to the case of RL-based cognitive modeling in HCI. We focus on five topics: (1) modeling human cognitive capabilities using RL, (2) model testing (e.g., parameter recovery checking), (3) creation of testing and training environments for RL agents, (4) parameter inference, and (5) validation against benchmark datasets in HCI. After completing this section, attendees will have necessary tools for model construction and validation.

Schedule and Registration

The course contains two online sessions in subsequent days. Presence in both sessions is required for passing the course.

April 26. 09:00-12:00 (New Orleans time), 14:00-17:00 (GMT)

  • Introduction to cognitive modeling

  • Deep learning

April 27. 09:00-12:00 (New Orleans time), 14:00-17:00 (GMT)

  • Reinforcement learning

  • Modeling workflow


In order to enroll to the course, please register to the CHI conference and add the workshop. This is done at https://web.cvent.com/event/39da8b29-3829-4548-829e-750fc9dd732e/ If you have already registered to CHI, you can modify the existing registration and add this course. For more instructions, please send an email to jussi.p.p.jokinen@jyu.fi

There are no specific skills required from the participants, but some familiarity with programming makes following the course easier.

Each course attendant must have registered to at least one day of the main conference.

Instructors

Jussi P.P. Jokinen

Assistant Professor, Cognitive Science

University of Jyväskylä

https://www.jyu.fi/it/fi/tiedekunta/henkilosto/henkilosto/jokinen-jussi

Antti Oulasvirta

Professor, Computer Science

Aalto University

http://users.comnet.aalto.fi/oulasvir/

Andrew Howes