Abstract: Multi-task learning ideally allows robots to acquire a diverse repertoire of useful skills. However, many multi-task reinforcement learning efforts assume the robot can collect data from all tasks at all times. In reality, the tasks that the robot learns arrive sequentially, depending on the user and the robot's current environment. In this work, we formalize a practical sequential multi-task RL problem that is motivated by the practical constraints of physical robotic systems. We analyze several of the design decisions for algorithms in such settings, and derive an approach that effectively leverages the data and policies learned for previous tasks to cumulatively grow the robot's skill-set. In a series of simulated robotic manipulation experiments, we find this approach accelerates learning of each additional task, requiring less than half the number of samples than learning each of the tasks from scratch, while avoiding impractical round-robin data collection. On a Franka Emika Panda robot arm, our approach can incrementally learn a policy for 6 challenging tasks, including bottle capping and block insertion.

A Franka Emika robot arm learns a sequence of manipulation tasks, including object insertion and bottle capping, by retaining experience from previously solved tasks. Our algorithm can learn a sequence of tasks with different physical setups and objectives with fewer samples compared to when each task is learned from scratch.