Imagine:
Language as a Cognitive Tool
to Imagine Goals in Curiosity Driven Exploration
Cédric Colas*, Tristan Karch*, Nicolas Lair*,
Jean-Michel Dussoux, Clément Moulin Frier, Peter Ford Dominey, Pierre-Yves Oudeyer
*equal contribution
Accepted at NeurIPS 2020
Code : Colab Notebook - Imagine Github repo - Playground Github repo
Abstract
Developmental machine learning studies how autonomous agents can model the way children learn open-ended repertoires of skills. Such agents need to create and represent goals, select which ones to pursue and learn to achieve them. Recent approaches have considered goal spaces that were either fixed and hand-defined or learned using generative models of states. This limited agents to sample goals within the distribution of known effects. We argue that the ability to imagine out-of-distribution goals is key to enable creative discoveries and open-ended learning. Children do so by leveraging the compositionality of language as a tool to imagine descriptions of outcomes they never experienced before, targeting them as goals during play. We introduce IMAGINE, an intrinsically motivated deep reinforcement learning architecture that models this ability. Such imaginative agents, like children, benefit from the guidance of a social peer who provides language descriptions. To take advantage of goal imagination, agents must be able to leverage these descriptions to interpret their imagined out-of-distribution goals. This generalization is made possible by modularity: a decomposition between learned goal-achievement reward function and policy relying on deep sets, gated attention and object-centered representations. We introduce the Playground environment and study how this form of goal imagination improves generalization and exploration over agents lacking this capacity. In addition, we identify the properties of goal imagination that enable these results and study the impacts of modularity and social interactions.
Imagine EXplained
Rollout of the agent for different goals
Goals communicated by the social partner
Navigation goals
The agent can target different areas of the 2D space.
Grasp goals
The agent can grasp any object it targets in the scene.
Grow animals with water
The agent can grow an animal by bringing it water.
Grow animals with food
This also works with food.
Imagined goals:
Grow plant like animals
The agent generalizes from growing animals to growing plants but it does not work with food.
Grow plant with water
The agent can adapt its behavior to bring more water to plants.
Grow furniture
The agent imagines crazy goals, like growing pieces of furniture, which does not work.
Going left-right
The agent also invents meaningless goals