Latent Action Priors for Locomotion with Deep Reinforcement Learning
Anonymous Author(s)
Code [TBA] Torque control deployment code [TBA] Paper [TBA]

Question: How to efficiently train DRL policies with minimal expert knowledge?

TL;DR. Latent action priors combined with a single style reward term accelerate locomotion learning without extensive reward tuning.

Abstract. Deep reinforcement learning (DRL) often leads to brittle, unnatural behaviors or requires extensive reward tuning. We propose a latent action space combined with a style reward term to accelerate locomotion learning without reward tuning. Both the latent action space and the imitation rewards are derived from a demonstrations. While the pre-trained latent action spaces alone improve the learned task reward and the sample efficiency for learning, especially the combination with a single imitation reward from the same demonstration is a powerful tool to obtain deployable locomotion behaviors.

Method. A latent action space for policy learning with DRL is trained via an autoencoder loss L on an expert demonstration. We combine the latent action space with a single style reward term. Both the decoder and style reward can be derived from a demonstration, such as an open-loop gait generator provided by an existing datasets or LLM response.

video_2.mp4

LLM-designed demonstration
with an open-loop controller in simulation

Learned policy with
user command inputs
(2x speed)

Generalization - still using
the same demonstration
(2x speed)

Learned Policy in challenging direct torque control mode

Variety of environments

We benchmark our approach in a variety of simulated environments based on existing demonstrations. First, we obtain the demonstration ...

Demonstration by open-loop controller (A. Raffin, 2021)

Demonstration by
SAC policy

Demonstration from
expert dataset (F. Al-Hafez, 2023)

... and then train a robust policy with a latent action prior and a single style reward term using PPO.

Learned policy

Latent action priors improve policy learning measured by task reward across all environments. Adding a single style reward term additionally improves the visual appearance of the gait in simulation for demonstrations generated with open-loop controllers.

Imitation

The combination of a latent action prior with a single style reward term leads to robust imitation of the demonstration.

Demonstration data
(F. Al-Hafez, 2023)

PPO
w/o reward tuning

PPO
+ latent action prior

PPO
+ latent action prior + style

Please refer to the paper for full results.

Latent Action Priors for Locomotion with Deep Reinforcement LearningAnonymous Author(s)Code [TBA] Torque control deployment code [TBA] Paper [TBA]

Latent Action Priors for Locomotion with Deep Reinforcement Learning
Anonymous Author(s)
Code [TBA] Torque control deployment code [TBA] Paper [TBA]