Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RL

Abstract

Hierarchical Reinforcement Learning (HRL) promises to solve long-horizon Reinforcement Learning (RL) tasks more efficiently than non-hierarchical counterparts by discovering and reusing temporally-extended skills. However, obtaining skills that are actually reusable remains an open challenge. Towards this end, we focus on abstractions that exploit the intuition of local dynamics: local transitions in different global contexts require similar kinds of action sequences. We build this into HRL through a contrastive representation learning method that aligns state-goal pairs with action-sequences in representation space. We then use embeddings from this method to train low-level skills, ensuring that they are reusable across local dynamics. The resulting algorithm CARL (Contrastive Action-based Representations for Reusable Local Control) shows both qualitative clustering of meaningful skills in complex humanoid environments and improved downstream performance through skill reuse in the OGBench benchmark.

Grid-World Evaluations

Below, we validate our hypothesis that our encoder captures a representation that can alias state-goal pairs that admit similar action sequences. We see that for 5 identical tasks that differ only in their raw x-y representation, a policy learned on top of our representation generalizes perfectly. In contrast, a policy learned on raw states fails to perform on any of the tasks on which it was not trained. The encoder that generates these representations is trained on all five tasks, while the policies are only trained on task 1 and then tested across all five tasks.

Humanoid Visualizations

We visualize the behaviors of 30 nearest neighbors using 3 methods: a random encoder, ACRO, and our method CARL. We choose diverse behaviors to illustrate the different kinds of skills CARL is able to find. These behaviors are captioned in each video. Compared to a random encoder, CARL and ACRO can find nearest neighbors exhibiting high behavioral similarity despite global state differences.  The similarities in ACRO and CARL suggest some shared formalism, which we discuss briefly in our paper.  We leave this to further work to fully resolve.

Random

CARL