Language as a Cognitive Tool
to Imagine Goals in Curiosity Driven Exploration
Cédric Colas*, Tristan Karch*, Nicolas Lair*,
Jean-Michel Dussoux, Clément Moulin Frier, Peter Ford Dominey, Pierre-Yves Oudeyer
Developmental machine learning studies how autonomous agents can model the way children learn open-ended repertoires of skills. Such agents need to create and represent goals, select which ones to pursue and learn to achieve them. Recent approaches have considered goal spaces that were either fixed and hand-defined or learned using generative models of states. This limited agents to sample goals within the distribution of known effects. We argue that the ability to imagine out-of-distribution goals is key to enable creative discoveries and open-ended learning. Children do so by leveraging the compositionality of language as a tool to imagine descriptions of outcomes they never experienced before, targeting them as goals during play. We introduce IMAGINE, an intrinsically motivated deep reinforcement learning architecture that models this ability. Such imaginative agents, like children, benefit from the guidance of a social peer who provides language descriptions. To take advantage of goal imagination, agents must be able to leverage these descriptions to interpret their imagined out-of-distribution goals. This generalization is made possible by modularity: a decomposition between learned goal-achievement reward function and policy relying on deep sets, gated attention and object-centered representations. We introduce the Playground environment and study how this form of goal imagination improves generalization and exploration over agents lacking this capacity. In addition, we identify the properties of goal imagination that enable these results and study the impacts of modularity and social interactions.
Rollout of the agent for different goals
Goals communicated by the social partner
The agent can target different areas of the 2D space.
The agent can grasp any object it targets in the scene.
Grow animals with water
The agent can grow an animal by bringing it water.
Grow animals with food
This also works with food.
Grow plant like animals
The agent generalizes from growing animals to growing plants but it does not work with food.
Grow plant with water
The agent can adapt its behavior to bring more water to plants.
The agent imagines crazy goals, like growing pieces of furniture, which does not work.
The agent also invents meaningless goals