PaCo: Parameter Compositional Multi-Task RL
Lingfeng Sun¹* Haichao Zhang²* Wei Xu² Masayoshi Tomizuka¹
¹University of California Berkeley ²Horizon Robotics
*Equal Contribution
NeurIPS 2022
[PDF] [OpenReview] [Code]
Abstract
The purpose of multi-task reinforcement learning (MTRL) is to train a single policy that can be applied to a set of different tasks. Sharing parameters allows us to take advantage of the similarities among tasks. However, the gaps between contents and difficulties of different tasks bring us challenges on both which tasks should share the parameters and what parameters should be shared, as well as the optimization challenges due to parameter sharing. In this work, we introduce a parameter-compositional approach (PaCo) as an attempt to address these challenges. In this framework, a policy subspace represented by a set of parameters is learned. Policies for all the single tasks lie in this subspace and can be composed by interpolating with the learned set. It allows not only flexible parameter sharing but also a natural way to improve training. We demonstrate the state-of-the-art performance on Meta-World benchmarks, verifying the effectiveness of the proposed approach.
PaCo Framework
In PaCo, the network parameter vector for a task (task parameter vector) is instantiated in a compositional form based on the shared base parameter set and the task-specific compositional vector. Then the networks are used in the standard way for generating actions or computing the loss. During training, the shared base parameter set will be impacted by all the task losses, while compositional vector is impacted by the corresponding task loss only.
Performance Comparison
The benchmark task is MetaWorld 10 with random goals (MT10-rand). For training on, 20M environment steps are used for the 10 tasks together (2M per task).
The performance of different methods are shown in the table on the left.
Visualization of Compositional Vectors
2D PCA projection of the ten 5D compositional vectors
(a) learned on MT10-rand task setting, and
(b) learned on MT10-fixed setting.
Visualization of Samples from Policy Subspace
(a) Compositional vectors in the space of a unit circle: learned reach, push, peg-inset-side policies (denoted with △) and sampled policies on the unit circle.
(b) Visualization (using the peg-insert-side task) of example policies: the learned peg-inset-side policy and two sampled ones.
PaCo Demo
A video of the the agent controlled by a single trained PaCo model to solve all 10 tasks from MetaWorld 10 (MT10) is shown on the right (full video is here).
Related Publications and Resources
PaCo: Parameter-Compositional Multi-Task Reinforcement Learning
Lingfeng Sun*, Haichao Zhang*, Wei Xu and Masayoshi Tomizuka
Advances in Neural Information Processing Systems (NeurIPS), 2022
[PDF] [OpenReview] [Code]