Abstract

Quality-Diversity (QD) algorithms are powerful exploration algorithms that allow robots to discover large repertoires of diverse and high-performing skills. However, QD algorithms are sample inefficient and require millions of evaluations. In this paper, we propose Dynamics-Aware Quality-Diversity (DA-QD), a framework to improve the sample efficiency of QD algorithms through the use of dynamics models. We also show how DA-QD can then be used for continual acquisition of new skill repertoires. To do so, we incrementally train a deep dynamics model from experience obtained when performing skill discovery using QD. We can then perform QD exploration in imagination with an imagined skill repertoire. We evaluate our approach on three robotic experiments. First, our experiments show DA-QD is 20 times more sample efficient than existing QD approaches for skill discovery. We then demonstrate learning an entirely new skill repertoire in imagination to perform zero-shot learning. Finally, we show how DA-QD is useful and effective for solving a long horizon navigation task and for damage adaptation in the real world.

Real-World Experiments

Planning and damage recovery with the learnt skill repertoires are performed using the RTE algorithm. Skill-space planning for a long-horizon navigation task is done using a Monte-Carlo Tree Search (MCTS) planner coupled with Gaussian Process models which enable sim-to-real transfer of the skills.

Despite requiring 20 times less evaluations, the quality of the skill repertoire is the same as one generated with a normal QD algorithm, and does not impact performance on longer-horizon navigation tasks using RTE.

DA-QD Skill Repertoire - No Damage Scenario

DA-QD Skill Repertoire - Damaged Scenario

Continual Acquisition of New Skill Repertoires