Manjie Xu1, 2, *, Guangyuan Jiang2, *, Wei Liang1, 3, ✉️, Chi Zhang4, ✉️ , Yixin Zhu2, ✉️
*equal contribution ✉️corresponding authors
1 School of Computer Science & Technology, Beijing Institute of Technology
2 Institute for AI, Peking University
3 Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing, China
4 National Key Laboratory of General Artificial Intelligence, BIGAI
NeurIPS23 Datasets and Benchmarks
Children quickly resolve uncertainty by generating hypotheses and testing them via active trials.
One of the fundamental cognitive abilities of humans is to quickly resolve uncertainty by generating hypotheses and testing them via active trials. Encountering a novel phenomenon accompanied by ambiguous cause-effect relationships, humans make hypotheses against data, conduct inferences from observation, test their theory via experimentation, and correct the proposition if inconsistency arises.
Reasoning under uncertainty is one of the hallmarks of human intelligence. From Hume's quest for causality to every modern scientific discovery, humans employ hypothesis testing to actively explore, collect new evidence, and reason to resolve the uncertainty — drawing knowns from unknowns. To date, machines still lag behind from how we, humans, even infants, understand the world, reason cause-and-effect relations, and make scientific discoveries.
Whether the intelligence we achieve today manages to perform such reasoning under uncertainty?
In this work, we devise the IVRE environment for evaluating artificial agents' reasoning ability under uncertainty. IVRE is an interactive environment featuring rich scenarios centered around Blicket detection. By evaluating modern artificial agents in IVRE, we notice a clear failure of today's learning methods compared to humans. Such inefficacy in interactive reasoning ability under uncertainty calls for future research in building humanlike intelligence.
Agents in IVRE are placed into environments with various ambiguous action-effect pairs and asked to figure out each object's role. Agents are encouraged to propose effective and efficient experiments to validate their hypotheses based on observations and gather more information. The game ends when all uncertainties are resolved or the maximum number of trials is consumed.
An online interactive environment for human participants is available.
You can also download and deploy your own IVRE following instructions in our repo.
Some example episodes are also available in case you only want to try without rendering by yourself.
Here we provide a simple walkthrough for the web version IVRE.
If you find IVRE useful, please cite us:
@inproceedings{xu2023interactive,
title={Interactive Visual Reasoning under Uncertainty},
author={Xu, Manjie and Jiang, Guangyuan and Liang, Wei and Zhang, Chi and Zhu, Yixin},
booktitle={NeurIPS},
year={2023}
}
