Exploration bonuses in reinforcement learning guide long-horizon exploration by defining custom intrinsic objectives. Several exploration objectives like count-based bonuses, pseudo-counts, and state-entropy maximization are non-stationary and hence are difficult to optimize for the agent. While this issue is generally known, it is usually omitted and solutions remain under-explored. The key contribution of our work lies in transforming the original non-stationary rewards into stationary rewards through an augmented state representation. For this purpose, we introduce the Stationary Objectives For Exploration (SOFE) framework. SOFE requires identifying sufficient statistics for different exploration bonuses and finding an efficient encoding of these statistics to use as input to a deep network. SOFE is based on proposing state augmentations that expand the state space but hold the promise of simplifying the optimization of the agent's objective. We show that SOFE improves the performance of several exploration objectives, including count-based bonuses, pseudo-counts, and state-entropy maximization. Moreover, SOFE outperforms prior methods that attempt to stabilize the optimization of intrinsic objectives. We demonstrate the efficacy of SOFE in hard-exploration problems, including sparse-reward tasks, pixel-based observations, 3D navigation, and procedurally generated environments.

Although optimizing for the same intrinsic reward distribution, SOFE allows for better optimization of the objective.

Our proposed augmentations achieve higher downstream task performance in the sparse reward settings

SOFE generally provides statistically significant improvements in sparse-reward tasks across RL algorithms and exploration modalities. Without exploration bonuses, agents fail to find the extrinsic rewards during training.

SOFE further improves the performance of the SOTA method for episodic exploration in procedurally generated environments

To understand how the policy uses the augmented information in the states, we craft artificial state visitation frequencies that indicate that all the states have been visited but a specific one (denoted with a red box in the Figure below). We find that augmented agents effectively make use of the additional information to direct their exploration toward the unvisited states.