We develop a new continual meta-learning method to address challenges in sequential multi-task learning. In this setting, the agent's goal is to achieve high reward over any sequence of tasks quickly. Prior meta-reinforcement learning algorithms have demonstrated promising results in accelerating the acquisition of new tasks. However, they require access to all tasks during training. Beyond simply transferring past experience to new tasks, our goal is to devise continual reinforcement learning algorithms that learn to learn, using their experience on previous tasks to learn new tasks more quickly. We introduce a new method, continual meta-policy search (CoMPS), that removes this limitation by meta-training in an incremental fashion, over each task in a sequence, without revisiting prior tasks. CoMPS continuously repeats two subroutines: learning a new task using RL and using the experience from RL to perform completely offline meta-learning to prepare for subsequent task learning. We find that CoMPS outperforms prior continual learning and off-policy meta-reinforcement methods on several sequences of challenging continuous control tasks.

We construct a continual meta learning method for efficient sequential multi-task ​learning

1. The algorithm uses reinforcement learning to interact with the world and collect experience on a single task. ​

2. The reinforcement learning policy starts with a set of policy parameters​

3. The agent trains these parameters on task k. During reinforcement learning training two collections of experience are kept, all experience and the best experience which is, the experience that achieved the highest average reward.​

4. This data is saved and stored for meta-training​

5. Our offline meta-self imitation method trains a policy to be capable of matching the performance of the best experience on prior tasks quickly.​

6. The policy parameters from the meta-self imitation learning process are used to initialize the RL policy for the next task.​

7. This makes collecting the best experience occur faster on the new task. This process repeats itself for each new task.

• Can CoMPS train without revisiting prior tasks?​

• How sample efficient is CoMPS?​

• Does learning efficiency increase as more tasks are solved?

• Introduced the sequential multi-task setting for reinforcement learning​

• This setting is more realistic for agents that can only interact with one task at a time.​

• A new Meta-RL algorithm that outperforms prior methods ​

  1. 1. 1. • Performance improves as more task are solved

The code to reproduce the experiments can be downloaded here