Active interaction with the environment is fundamental to human understanding of the world around us. Both neural and behavioral studies indicate that through active engagement with their surroundings, humans garner critical insights and foster a profound understanding of complex phenomena. When confronted with partial or ambiguous data, our innate response is to seek supplementary evidence, hypothesize, and put forth possible explanations, sometimes even reevaluating initial assumptions. This iterative process persists until a satisfactory resolution emerges.

We present 🔍Conan to capture the dynamic and exploratory essence of abductive reasoning—termed herein as active reasoning.

At its core, 🔍Conan is conceived as a detective game, transmuted into a question-answering challenge. Here, the detective is tasked with a query and an “incident scene” riddled with traces left by a vandal. Given the initial paucity of conclusive information, the detective must embark on an in-depth exploration of the scene. As the inquiry progresses, the detective has the opportunity to actively scout its environment, continually reshaping and honing its hypotheses, especially when new revelations potentially contradict the prior hypothesis.We meticulously craft questions within        🔍Conan to span various levels of abstraction, from localized intentions (Intent) to overarching objectives (Goal) and survival states (Survival).

Building on previous work that utilizes hierarchical models for task decomposition, our pipeline is structured into two primary phases: an exploration phase for trace collection, followed by an abductive reasoning phase. Initially, interaction with the playground is carried out to collect relevant visual information, which is subsequently leveraged in the reasoning phase to infer answers to the posed questions.

Computationally, our pipeline first employs reinforcement learning agents as explorers that learn an exploration policy based on the traces and the question, thereby rendering it goal-oriented. Next, given the question, we recruit vision-language models to predict the answer based on the observation. A key-frame extractor is inserted into the two phases to selectively identify relevant frames for abduction. The individual components undergo separate training procedures.

In the standard setting, we discern that while models exhibit some aptitude in tackling low-level Intent questions, they struggle with higher-level questions pertaining to Goal and Survival. Among the models, Flamingo-Mini ascends to the pinnacle. FrozenBiLM models also perform relatively well. The DeBERTa variant slightly outperforms BERT, insinuating that a robust language backbone can improve general comprehension.

With the Ideal Explorer, we notice a clear performance boost across all tasks, particularly in the Goal and Survival questions. These results allude to the potential bottlenecking of models’ abductive reasoning capability due to the insufficient information collected, underscoring the significance of effective exploration. Remarkably, the Vanilla-Trans exhibits the greatest increase, insinuating that, in comparison to other baseline models, it is markedly vulnerable to insufficient evidence.

For AfD results, nearly all multimodal models exhibit performance on par with end-to-end supervisedly trained models. Remarkably, FrozenBiLM models even surpass the performance observed in standard settings. Examining task-specific results, a notable performance uplift in the Survival task models is discernible for almost all models relative to the standard setting, albeit sharing the same observation. These results intimate that the inclusion of deductive information sensitizes the detective to vandal’s concerns during task execution.

If you find 🔍Conan useful, please cite us:

@inproceedings{xu2023conan,

  title={Conan: Active Reasoning in an Open-World Environment},

  author={Xu, Manjie and Jiang, Guangyuan and Liang, Wei and Zhang, Chi and Zhu, Yixin},

  booktitle={NeurIPS},

  year={2023}

}