BILE: An Effective Behavior-based Latent Exploration Scheme for Deep Reinforcement Learning
Yiming Wang¹, Kaiyan Zhao², Yan Li³, Leong Hou U¹
¹State Key Laboratory of Internet of Things for Smart City, University of Macau
²School of Computer Science, Wuhan University
³School of Artificial Intelligence, Shenzhen Polytechnic University
To Be Updated
Abstract
Efficient exploration of state spaces is critical for the success of deep reinforcement learning (RL). While many methods leverage exploration bonuses to encourage exploration instead of relying solely on extrinsic rewards, these bonus-based approaches often face challenges with learning efficiency and scalability, especially in environments with high-dimensional state spaces.
To address these issues, we propose BehavIoral metric-based Latent Exploration (BILE). The core idea is to learn a compact representation within the behavioral metric space that preserves value differences between states. By introducing additional rewards to encourage exploration in this latent space, BILE drives the agent to visit states with higher value diversity and exhibit more behaviorally distinct actions, leading to more effective exploration of the state space. Additionally, we present a novel behavioral metric for efficient and robust training of the state encoder, backed by theoretical guarantees.
Visualization of heatmap of trajectories from different algorithms
RND
RIDE
LIBERTY
BILE