CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions

Abstract

Recent advances in Artificial Intelligence and deep learning have revived the interest in studying the gap between the reasoning capabilities of humans and machines. In this ongoing work, we introduce CRAFT, a new visual question answering dataset that requires causal reasoning about physical forces and object interactions. It contains 38K video and question pairs that are generated from 3K videos from 10 different virtual environments, containing different number of objects in motion that interact with each other. Two question categories from CRAFT include previously studied descriptive and counterfactual questions. Besides, inspired by the theory of force dynamics from the field of human cognitive psychology, we introduce new question categories that involve understanding the intentions of objects through the notions of cause, enable, and prevent. Our preliminary results demonstrate that even though these tasks are very intuitive for humans, the implemented baselines could not cope with the underlying challenges.

Example simulations and questions

Overview

Descriptive
Q: “How many objects fall to the ground?” A: “2”Q: “After entering the basket, does the small yellow square collide with other objects?” A: “True”
CounterfactualQ: “How many objects fall to the ground, if the small yellow box is removed?” A: “1”Q: “Does the small gray box enter the basket, if any other single one of the objects is removed?” A: “True”
CauseQ: “Does the small brown sphere cause the tiny yellow box to enter the basket?” A: “True”Q: “Does the small gray block cause the big cyan triangle to collide with the basket?” A: “True”
EnableQ: “Does the small brown sphere enable the small yellow box to enter the basket?” A: “False”Q: “How many objects does the small gray block enable to enter the basket?” A: “0”
PreventQ: “Does the small yellow square prevent the tiny brown circle from entering the basket?” A: “True”Q: “How many objects does the large cyan triangle prevent from entering the basket?” A: “1”

Descriptive

Q: How many cyan objects collide with the bucket?A: 2
Q: What is the color of object that the tiny cyan circle first collides with?A: blue
Q: After falling into the bucket, does the large blue block collide with other objects?A: True
Q: How many objects collide with the container?A: 3
Q: What is the color of object that the big green circle first collides with?A: brown
Q: After entering the bucket, does the small brown cube collide with other objects?A: True
Q: Before getting into the container, does the large green circle collide with other objects?A: True
Q: What is the color of first object to collide with the big gray circle?A: purple
Q: What shape is the object that the small green triangle last collides with?A: cube
Q: After falling into the container, does the large gray circle collide with other objects?A: False
Q: Before hitting the bottom, does the tiny cyan triangle collide with other objects?A: False

Counterfactual

Q: Does the small green circle hit the ground, if the large gray block is removed?A: True
Q: Does the tiny green circle fall to the ground, if any other single one of the objects is removed?A: True
Q: If any other single one of the objects is removed, does the tiny green circle collide with the basket?A: False


Q: If the small gray circle is removed, does the small red triangle get into the bucket?A: False
Q: If any other single one of the objects is removed, does the small gray circle fall to the bottom?A: True
Q: If the big gray circle is removed, does the tiny brown block collide with the basket?A: False
Q: If any other single one of the objects is removed, does the big gray circle enter the basket?A: True
Q: Does the big gray circle collide with the container, if any other single one of the objects is removed?A: True

Enable

Q: Does the big yellow circle enable the small blue triangle to collide with the basket?A: True

Q: There is a large red circle, does it enable the small gray circle to collide with the ground?A: True

Q: Does the large cyan circle allow the collision between the large green circle and the bottom?A: True

Prevent

Q: Does the small brown circle hinder the small cyan circle from entering the bucket?A: True

Q: Is the small blue circle is held by the big yellow circle from colliding with the container?A: True

Q: Does the tiny green circle keep the small brown circle from falling into the container?A: True
Q: Is the tiny brown circle is blocked by the small green circle from colliding with the container?A: True

Cause

Q: There is a tiny gray triangle, does it stimulate the tiny brown block to go into the container?A: True
Q: Does the large purple circle stimulate the collision between the big cyan triangle and the bottom?A: True
Q: There is a big purple circle, does it cause the tiny cyan cube to end up in the bucket?A: True