Skill Transfer via Partially Amortized Hierarchical Planning
Kevin Xie*, Homanga Bharadhwaj*, Danijar Hafner, Animesh Garg, Florian Shkurti
International Conference on Learning Representations (ICLR) 2021
*Kevin and Homanga contributed equally.
To quickly solve new tasks in complex environments, intelligent agents need to build up reusable knowledge. For example, a learned world model captures knowledge about the environment that applies to new tasks. Similarly, skills capture general behaviors that can apply to new tasks. In this paper, we investigate how these two approaches can be integrated into a single reinforcement learning agent. Specifically, we leverage the idea of partial amortization for fast adaptation at test time. For this, actions are produced by a policy that is learned over time while the skills it conditions on are chosen using online planning. We demonstrate the benefits of our design decisions across a suite of challenging locomotion tasks and demonstrate improved sample efficiency in single tasks as well as in transfer from one task to another, as compared to competitive baselines.
To quickly solve new tasks in complex environments, intelligent agents need to build up reusable knowledge. For example, a learned world model captures knowledge about the environment that applies to new tasks. Similarly, skills capture general behaviors that can apply to new tasks. In this paper, we investigate how these two approaches can be integrated into a single reinforcement learning agent. Specifically, we leverage the idea of partial amortization for fast adaptation at test time. For this, actions are produced by a policy that is learned over time while the skills it conditions on are chosen using online planning. We demonstrate the benefits of our design decisions across a suite of challenging locomotion tasks and demonstrate improved sample efficiency in single tasks as well as in transfer from one task to another, as compared to competitive baselines.
Video Skill Visualizations
Video Skill Visualizations
We select arbitrary skills from skill space and fix them throughout 100 environment steps. The videos below show that the policy learns meaningful and distinct behaviours conditioned on the skill. Note that the agents are fully deterministic and variations are only due to the skill conditioning.
We select arbitrary skills from skill space and fix them throughout 100 environment steps. The videos below show that the policy learns meaningful and distinct behaviours conditioned on the skill. Note that the agents are fully deterministic and variations are only due to the skill conditioning.
Quadruped Walk: Skills Within the Unit Ball
Quadruped Walk: Skills Within the Unit Ball
Quadruped Walk: Skills Outside the Unit Ball
Quadruped Walk: Skills Outside the Unit Ball
Transfer to Quadruped Reach
Transfer to Quadruped Reach
We transfer the learned low level skills to a target reaching task. The red spot represents the target and is randomly sampled in the arena.
We transfer the learned low level skills to a target reaching task. The red spot represents the target and is randomly sampled in the arena.
Random Obstacle Walk Pretransfer Task
Random Obstacle Walk Pretransfer Task
The agent is pretrained to walk at constant speed around random obstacle configuration in the arena.
Cove Obstacle Reach Target Transfer Target Task
Cove Obstacle Reach Target Transfer Target Task
Dreamer after 80 episodes
Dreamer after 80 episodes
LSP after 80 episodes
LSP after 80 episodes
Dreamer transfers the world model, policy and value function. LSP transfers the world model subskill policy and value function.