In this work, we focus on the combination of analytical and learning-based techniques to help researchers solve challenging robot locomotion problems. Specifically, we explore the combination of curricular trajectory optimization (CTO) and deep reinforcement learning (RL) for quadruped hurdling tasks. Our goal is to provide a framework for engineers and researchers to combine the generalization capabilities of feedback policies such as neural networks with the efficiency of trajectory optimization. We use the trajectories generated from optimization as an imitation learning supervisor to provide an additional gradient signal to the RL algorithm. To generate these trajectories, we introduce a curricular optimization algorithm where a discrete set of increasingly difficult tasks are solved via black-box trajectory optimization where each task is initialized by parameters obtained from simpler tasks. We evaluate our approach on various robot hurdling tasks where the robot needs to jump over an obstacle of varying size and location. Our results show that we achieve greater sample efficiency than state-of-the-art reinforcement learning when solving the task, and significantly greater performance than the original trajectories.

Left: CMA-ES is used to generate reference trajectories for M different environments. These trajectories are stored in the trajectory buffer.

Right: Reference trajectories are sampled from the buffer and used for the imitation reward. The RL agent uses the combined imitation and task rewards to update its parameters and find the optimal policy.