IVRE: Interactive Visual REasoning under Uncertainty

Manjie Xu1, 2, *, Guangyuan Jiang2, *, Wei Liang1, 3, ✉️, Chi Zhang4, ✉️ , Yixin Zhu2, ✉️ 

 *equal contribution  ✉️corresponding authors


1 School of Computer Science & Technology, Beijing Institute of Technology 

2 Institute for AI, Peking University

3 Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing, China

4 National Key Laboratory of General Artificial Intelligence, BIGAI

NeurIPS23 Datasets and Benchmarks

Children quickly resolve uncertainty by generating hypotheses and testing them via active trials. 

One of the hallmarks of human intelligence

One of the fundamental cognitive abilities of humans is to quickly resolve uncertainty by generating hypotheses and testing them via active trials. Encountering a novel phenomenon accompanied by ambiguous cause-effect relationships, humans make hypotheses against data, conduct inferences from observation, test their theory via experimentation, and correct the proposition if inconsistency arises. 

Reasoning under uncertainty is one of the hallmarks of human intelligence. From Hume's quest for causality to every modern scientific discovery, humans employ hypothesis testing to actively explore, collect new evidence, and reason to resolve the uncertainty — drawing knowns from unknowns. To date, machines still lag behind from how we, humans, even infants, understand the world, reason cause-and-effect relations, and make scientific discoveries.

The IVRE benchmark

Whether the intelligence we achieve today manages to perform such reasoning under uncertainty?

In this work, we devise the IVRE environment for evaluating artificial agents' reasoning ability under uncertainty. IVRE is an interactive environment featuring rich scenarios centered around Blicket detection. By evaluating modern artificial agents in IVRE, we notice a clear failure of today's learning methods compared to humans. Such inefficacy in interactive reasoning ability under uncertainty calls for future research in building humanlike intelligence.

Agents in IVRE are placed into environments with various ambiguous action-effect pairs and asked to figure out each object's role. Agents are encouraged to propose effective and efficient experiments to validate their hypotheses based on observations and gather more information. The game ends when all uncertainties are resolved or the maximum number of trials is consumed.



Resources

An online interactive environment for human participants is available.

You can also download and deploy your own IVRE following instructions in our repo.

Some example episodes are also available in case you only want to try without rendering by yourself.

Here we provide a simple walkthrough for the web version IVRE.

Control Button Click New Game to start a new game, Exit to exit the current game. 

Info Key info of the current game, including the panel count, reward for last action and the total reward of the current game.

Current Panel The current panel of the selected objects and the blicket machine.

History Panel History panels of all of the past trials. Four context panels and several trial panels. Can be used to help do reasoning and propose new trials.

Belief/Trial We show all possible blickets in the current game here. Select blickets in your mind  according to your current beliefs when “Belief  is required.  Make your own trial when Trial  is required. Click Submit to submit.

Citation

If you find IVRE useful, please cite us:


@inproceedings{xu2023interactive,

  title={Interactive Visual Reasoning under Uncertainty},

  author={Xu, Manjie and Jiang, Guangyuan and Liang, Wei and Zhang, Chi and Zhu, Yixin},

  booktitle={NeurIPS},

  year={2023}

}