Abstracting Robot Manipulation Skills via Mixture-of-Experts Diffusion Policies

Diffusion-based policies have recently shown strong results in robot manipulation, but their extension to multi-task scenarios is hindered by the high cost of scaling model size and demonstrations. We introduce Skill Mixture-of-Experts Policy (SMP), a diffusion-based mixture-of-experts policy that learns a compact orthogonal skill basis and uses sticky routing to compose actions from a small, task-relevant subset of experts at each step. A variational training objective supports this design, and adaptive expert activation at inference yields fast sampling without oversized backbones. We validate SMP in simulation and on a real dual-arm platform with multi-task learning and transfer learning tasks, where SMP achieves higher success rates and markedly lower inference cost than large diffusion baselines. These results indicate a practical path toward scalable, transferable multi-task manipulation: learn reusable skills once, activate only what is needed, and adapt quickly when tasks change. 

Overview of SMP.  Top:  Bimanual rollout of “put card in drawer” with key steps (1)–(5). Middle:  Skill decomposition by arm and phase: the state-adaptive orthonormal skill basis and sticky routing yields spatial specialization (left/right), and  organizes behavior into pick, adjust, reach, and release with corresponding end-effector (EE) actions. Bottom: Gate values over time show sparse, phase-consistent activation—only a few experts are active per step for efficient sampling.