Explore to Learn: Latent Exploration through Disentangled Synergy Patterns for Reinforcement Learning in Overactuated Control

Yiming Wang¹, Kaiyan Zhao², Xu Li¹, Yan Li³, Jiayu Chen⁴, Steven Morad¹, Leong Hou U¹

¹State Key Laboratory of Internet of Things for Smart City, University of Macau

²School of Computer Science, Wuhan University

³School of Artificial Intelligence, Shenzhen Polytechnic University

⁴Department of Data and Systems Engineering, University of Hong Kong

PDF

Code

Bibtex

Abstract

Control in high-dimensional action spaces remains a fundamental challenge in reinforcement learning (RL), primarily due to inefficient exploration of the action space.

While recent methods attempt to guide exploration, they often fall short of achieving the agility and coordination exhibited in biological motor control. Inspired by how organisms exploit muscle synergies for efficient movement, we propose Explore to Learn (ETL), a two-stage framework that first discovers fundamental synergy patterns and then leverages them for task-specific policy learning.

In the first stage, ETL discovers underlying synergy patterns by deploying a targeted exploration policy. These patterns are modeled as latent directions in a low-dimensional space, along which the agent is guided to collect diverse and structured muscle activation trajectories. A variational autoencoder (VAE) is then trained to encode high-dimensional actions into a latent space whose dimensions correspond to the synergy patterns.

In the second stage, the policy is trained entirely in this synergy-aware latent space, producing synergy coefficients that the decoder maps back to full-dimensional muscle actions. This structured representation significantly reduces the complexity of learning, while the decoder is further fine-tuned to enhance expressiveness and generalization across downstream tasks.

Motivation

Biological motor control leverages low dimensional muscle synergies for efficient coordination. Inspired by this principle, ETL learns disentangled synergy patterns (e.g., motor primitives like back kick, knee down, leg raise and leg swing) to reduce the burden of overactuated control, enabling tractable and structured exploration.

Overview

Overview of ETL (Explore to Learn) framework. ETL operates in two stages: Explore and Learn. In the exploration stage (left), a latent exploration policy is used to sample diverse synergy patterns, modeled as directions in a low-dimensional latent space. These patterns produce behaviorally distinct muscle activations, which are stored in the replay buffer. A Gaussian Mixture VAE (top right) is then trained to encode these trajectories into compact, disentangled synergy representations. In the learning stage (bottom right), a policy network operates in the learned latent space, outputting synergy coefficients that are decoded into full muscle activations. The decoder is further fine-tuned during policy training to enhance control expressiveness.

Bibtex

@inproceedings{wang2025explore,

title={Explore to Learn: Latent Exploration through Disentangled Synergy Patterns for Reinforcement Learning in Overactuated Control},

author={Yiming Wang and Kaiyan Zhao and Xu Li and Yan Li and Jiayu Chen and Steven Morad and Leong Hou U},

booktitle={The Fortyth Annual AAAI Conference on Artificial Intelligence},

year={2026}

}