CLEVRER-Humans: Describing Physical and Causal Events the Human Way

 Jiayuan Mao*, Xuelin Yang*,  Xikun Zhang, Noah D. Goodman, Jiajun Wu

∗indicates equal contribution. Authors ordered alphabetically.


Building machines that can reason about physical events and their causal relationships is crucial for flexible interaction with the physical world. However, most existing physical and causal reasoning benchmarks are exclusively based on synthetically generated events and synthetic natural language descriptions of the causal relationships. This design brings up two issues. First, there is a lack of diversity in both event types and natural language descriptions; second, causal relationships based on manually-defined heuristics are different from human judgments. To address both shortcomings, we present the CLEVRER-Humans benchmark, a video reasoning dataset for causal judgment of physical events with human labels. We employ two techniques to improve data collection efficiency: first, a novel iterative event cloze task to elicit a new representation of events in videos, which we term Causal Event Graphs (CEGs); second, a data augmentation technique based on neural language generative models. We convert the collected CEGs into questions and answers to be consistent with prior work. Finally, we study a collection of baseline approaches for CLEVRER-Humans question-answering, highlighting great challenges set forth by our benchmark.



  title={CLEVRER-Humans: Describing Physical and Causal Events the Human Way},

  author={Mao, Jiayuan and Yang, Xuelin and Zhang, Xikun and Goodman, Noah and Wu, Jiajun},

  booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track}



This work is partly supported by the Stanford Institute for Human-Centered AI (HAI), the Samsung Global Research Outreach (GRO) Program, ONR MURI N00014-22-1-2740, and Adobe, Amazon, Analog, Bosch, IBM, JPMC, Meta, and Salesforce. We also thank Keyi Hu for her contribution.