Learning More With Less: Sample-Efficient Model-Based RL for Loco-Manipulation

Abstract

By combining the agility of legged locomotion with the capabilities of manipulation, loco-manipulation platforms have the potential to perform complex tasks in real-world applications. To this end, state-of-the-art quadrupeds with manipulators, such as the Boston Dynamics Spot, have emerged to provide a capable and robust platform. However, the complexity of loco-manipulation control, as well as the black-box nature of commercial platforms, pose challenges for deriving accurate dynamics models and control policies. 

To address these challenges, we develop a hand-crafted kinematic model of a quadruped-with-arm platform which – employing recent advances in Bayesian Neural Network (BNN)–based learning – we leverage as a physical prior to efficiently learn an accurate dynamics model from limited data. We then derive control policies for loco-manipulation via model-based reinforcement learning (RL). We demonstrate the effectiveness of our approach on hardware using the Boston Dynamics Spot, accurately performing dynamic end-effector trajectory tracking even in low data regimes.

Evaluation

We demonstrate the effectiveness of our approach on the Boston Dynamics Spot, a state-of-the-art quadruped robot equipped with an arm for manipulation. We leverage our learned dynamics model to develop control policies for loco-manipulation via model-based RL and then use the learned policy to dynamically track two shapes with the robot’s end-effector: an ellipse and a helix.

We compare our main approach, SIM-FSVGD, to the two baseline models SIM-MODEL and FSVGD:

• SIM-MODEL: Hand-crafted kinematic model from first principles with learnable parameters.

• FSVGD: BNN-model that uses function space particle optimization and is widely applied in bayesian deep learning [2].

• SIM-FSVGD: BNN-model based on FSVGD that uses our kinematic model as a physical prior to achieve better sample efficiency [1].

[1] J. Rothfuss, et al., “Bridging the Sim-to-Real Gap with Bayesian Inference,” 2024

[2] Z. Wang, et al., “Function Space Particle Optimization for Bayesian Neural Networks,” 2019