Abstract:

We aim to control a robot to physically behave in the real world following any high-level language command like "cartwheel" or "kick ". Although human motion datasets exist, this task remains particularly challenging since generative models can produce physically unrealistic motions, which will be more severe for robots due to different body structures and physical properties. In addition, to control a physical robot to perform a desired motion, a control policy must be learned. We develop LAnguage-Guided mOtion cONtrol (LAGOON), a multi-phase method to generate physically realistic robot motions under language commands. LAGOON first leverages a pretrained model to generate human motion from a language command. Then an RL phase is adopted to train a control policy in simulation to mimic the generated human motion. Finally, with domain randomization, we show that our learned policy can be successfully deployed to a quadrupedal robot, leading to a robot dog that can stand up and wave its front legs in the real world to mimic the behavior of a hand-waving human. 

LAGOON leverages the benefits from both end-to-end generation and RL training. 

LAGOON first adopts a motion diffusion model to generate a human motion from the language description. The generated human motion is mapped to a robot body to create a semantically desired but physically unrealistic target robot motion. Then an RL phase is performed to learn a policy in a physics engine to control a robot to mimic the target motion. Finally, with domain randomization, we show that the learned RL policy can be deployed to a real-world robot