Rethinking Exploration in Reinforcement Learning with Effective Metric-Based Exploration Bonus
Conference on Neural Information Processing Systems (NeurIPS) Spotlight, 2024
Yiming Wang¹, Kaiyan Zhao¹², Furui Liu³, Leong Hou U¹
¹State Key Laboratory of Internet of Things for Smart City, University of Macau
²School of Computer Science, Wuhan University
³Zhejiang Lab
Abstract
Enhancing exploration in reinforcement learning (RL) through the incorporation of intrinsic rewards, specifically by leveraging state discrepancy measures within various metric spaces as exploration bonuses, has emerged as a prevalent strategy to encourage agents to visit novel states. The critical factor lies in how to quantify the difference between adjacent states as novelty for promoting effective exploration.
Nonetheless, existing methods that evaluate state discrepancy in the latent space under $L_1$ or $L_2$ norm often depend on count-based episodic terms as scaling factors for exploration bonuses, significantly limiting their scalability. Additionally, methods that utilize the bisimulation metric for evaluating state discrepancies face a theory-practice gap due to improper approximations in metric learning, particularly struggling with hard exploration tasks. To overcome these challenges, we introduce the Effective Metric-based Exploration-bonus (EME). EME critically examines and addresses the inherent limitations and approximation inaccuracies of current metric-based state discrepancy methods for exploration, proposing a robust metric for state discrepancy evaluation backed by comprehensive theoretical analysis. Furthermore, we propose the diversity-enhanced scaling factor integrated into the exploration bonus to be dynamically adjusted by the variance of prediction from an ensemble of reward models, thereby enhancing exploration effectiveness in particularly challenging scenarios.
Motivation
Previous exploration methods struggle with scalability issue especially in environments with high-dimensional states
curiosity-driven method
state-difference based method
our method
Overview
In this work, we identify the limitations of existing state discrepancy-based exploration methods: the reliance on episodic count-based scaling factor and theory-practice approximation gap, which leads to limited scalability, especially for hard exploration tasks within realistic environments. To address the issues, we summarize our contributions as follows:
We propose the Effective Metric-based Exploration-bonus (EME), which addresses the inherent limitations by proposing a robust metric for state discrepancy evaluation backed by comprehensive theoretical analysis: convergence guarantee and guaranteed value bound.
Furthermore, we propose the diversity-enhanced scaling factor integrated into the exploration bonus to enhance exploration effectiveness in particularly challenging scenarios.
Extensive experiments on hard exploration tasks from continuous control, discrete-action games and realistic environments have demonstrated the effectiveness and scalability of our method.
Bibtex
@inproceedings{wang2024rethinking,
title={Rethinking Exploration in Reinforcement Learning with Effective Metric-Based Exploration Bonus},
author={Yiming Wang and Kaiyan Zhao and Furui Liu and Leong Hou U},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024}
}