Redeeming Intrinsic Rewards via Constrained Optimization

Reinforcement learning algorithms work well on tasks where dense extrinsic rewards provide frequent feedback. This is the case for ATARI games like James Bond. Other games like Montezuma's Revenge have sparse extrinsic rewards that are difficult to find using random exploration schemes like epsilon-greedy, which prevents standard RL algorithms from learning a successful policy. Supplementing a sparse extrinsic reward with a dense intrinsic reward that incentivizes the agent to explore novel states can greatly improve performance. In any game where extrinsic rewards are not completely sparse however, intrinsic rewards might distract the agent from accomplishing the task at hand. The unpredictable effect of intrinsic rewards on performance make them difficult to apply broadly in practice. Our method Extrinsic-Intrinsic Policy Optimization (EIPO) dynamically tunes the balance between extrinsic and intrinsic rewards during training, and closes the performance gap between intrinsic and extrinsic reward optimization across environments.

EIPO eliminates the need to manually tune the balance between intrinsic and extrinsic rewards. To demonstrate this, we manually tune the intrinsic-extrinsic scaling coefficient lambda in several games where RND substantially underperforms the extrinsic only (EO) baseline. We see that lambda can have a big effect on performance. EIPO matches the best baseline in all environments, suggesting that our method can automatically tune the balance between intrinsic and extrinsic rewards as training progresses. In some environments (e.g. Star Gunner, Yars Revenge), EIPO significantly outperforms all baselines. These results show that dynamic tuning not only . We hope EIPO will encourage practitioners to use intrinsic rewards to improve exploration on new benchmark tasks, without worrying about hurting performance or manually tuning sensitive hyperparameters.