ERR@HRI 3.0: Multimodal Detection of Errors and Anticipation in Human-Robot Interactions
When: October 5, 2026
ICMI '26, Naples, Italy
When: October 5, 2026
ICMI '26, Naples, Italy
Register as a participant in this challenge in the Registration tab (available March 15)
As robots become increasingly integrated into human environments, their ability to detect and respond to errors remains critical for maintaining user trust and interaction quality. While recent advances in machine learning have improved error detection capabilities, most approaches are limited to specific contexts, controlled settings, or pre-extracted features, limiting their generalizability and applicability to real-world conditions. To address this challenge, the third edition of the ERR@HRI Challenge (ERR@HRI 3.0) provides researchers with two complementary datasets that enable end-to-end innovation in methods for both detecting and preventing errors in human-robot interaction.
The challenge offers raw, non-anonymized video data from naturalistic settings: (1) the Bystander Affect Detection (BAD) dataset, containing webcam recordings of 45 participants' spontaneous reactions to robot and human failure scenarios; and (2) the Bad Idea dataset, featuring 29 participants' anticipatory facial responses while predicting action outcomes before failures occur. Both datasets were collected via crowdsourcing, capturing the inherent variability of real-world conditions - diverse lighting, camera angles, participant positioning, and environmental contexts. This naturalistic variability, while challenging, provides an authentic testbed for developing robust error detection systems.
Participants are invited to develop machine learning models that can generalize across these diverse contexts and temporal stages. Submissions will be evaluated on standard classification metrics (e.g. F1-score) with consideration for real-world deployment constraints. This challenge is a step toward developing robust error detection and prevention systems that can operate in the variable conditions of real-world human-robot collaboration.
Registration opening: March 15, 2026
Training and development sets available: March 15, 2026
Baseline code and evaluation scripts available: May 1, 2026
Test sets available: June 1, 2026
Final code and results submission: June 8, 2026
Paper submission deadline: June 15, 2026
Notification of acceptance: July 8, 2026
Camera-ready paper: August 2, 2026 (hard deadline)
Challenge day: October 5, 2026
The ERR@HRI 3.0 Challenge will consist of three sub-challenges:
Track 1: Bystander Reaction Detection (BAD Dataset). Binary classification of whether the participant is observing a failure (vs. control scenario).
Track 2: Anticipatory Response Prediction (Bad Idea Dataset). Binary classification of participants’ predicted outcome (well/poorly) from their anticipatory behavior.
Cross-Dataset Generalization (Optional). Participants are encouraged to explore transfer learning and generalization across datasets, e.g. train on BAD dataset and test on Bad Idea dataset.
Information about the dataset can be found at this page. Training and development sets to be made public on March 15th.
TBA
Participants should use the training and validation set to develop their detection models, and submit the final models via email to errathri@gmail.com and mb2554@cornell.edu (in Cc). The final results will be evaluated by organizers on the test set. All participants will be ranked based on the results on the test set. Please look at this page for more details.
The challenge participants will be invited to submit a workshop-style paper describing their ML solutions and results on the dataset -- these will be peer-reviewed and once accepted, will appear in the 28th ACM International Conference on Multimodal Interaction Challenge/Workshop Proceedings. The format of the paper follows the same requirements as the main conference of the ACM ICMI 2026.
We will invite the top-3 performing teams to submit a paper. Instructions about the paper submission system will be announced soon.
TBA
The second edition (ERR@HRI2.0) of this challenge was organised in conjunction with the ACM International Conference on Multimedia 2025 (ACMMM) in Dublin, Ireland. The corresponding website can be found here.
The first edition (ERR@HRI) of this challenge was organised in conjunction with the ACM International Conference on Multimodal Interaction 2024 (ICMI) in San Josè, Costa Rica. The corresponding website can be found here.
Maria Teresa Parreira, mb2554@cornell.edu (via email)