Brandon Trabucco, Carnegie Mellon University, Pittsburgh

Abstract:

The standard approach to reinforcement learning is to fit policies and value functions specialized to a single agent design. This specialization precludes transferring existing models to agents with new designs that differ from the current one. The effect is that when parts on a robot break, or that robot is due for an upgrade, a new model must typically be trained from scratch, which is impractical for both the researcher and the end user. In this talk, I discuss the challenge of learning models that generalize beyond agents with the same design, and present our recent work on learning transferable policies by inferring agent morphology. Prior work has either assumed the physical connectivity of the agent (ie, morphology) is known in advance, or ignores this information. Instead, we propose to consider morphology a feature to be inferred during training, and show this relaxes assumptions in prior work and improves performance. The key idea behind our approach is to encode morphology as a sequence of discrete tokens with learned token representations, and process them with a Transformer-based policy. Our results on a standard benchmark suggest our Transformer generalizes better than current methods, and benefits from large datasets containing many unique agent designs.