Long horizon robot learning tasks with sparse rewards pose a significant challenge for current reinforcement learning algorithms.

A key feature enabling humans to learn challenging control tasks is that they often receive expert intervention that enables them to understand the high-level structure of the task before mastering low-level control actions.

We propose a framework for leveraging expert intervention to solve long-horizon reinforcement learning tasks. We consider option templates, which are specifications encoding a potential option that can be trained using reinforcement learning. We formulate expert intervention as allowing the agent to execute option templates before learning an implementation. This enables them to use an option, before committing costly resources to learning it.

We evaluate our approach on three challenging reinforcement learning problems, showing that it outperforms state-of-the-art approaches by two orders of magnitude.

Child development researchers placed 'sticky mittens' (mittens with velcro) on infants too young to actually grasp objects. The mittens allowed the infants to snag Velcro-fitted toys merely by swiping at them. In comparisons with infants who hadn't used the mittens, found the psychologists, those who had used the mittens subsequently showed more sophisticated abilities to explore objects.

The sticky mittens experiment exposed infants to the consequences of grasping before they actually learned how to grasp!