Continuously Discovering Novel Strategies via Reward-Switching Policy Optimization

Abstract

We present a novel paradigm, Reward-Switching Policy Optimization (RSPO), to discover diverse strategies in complex RL environments by iteratively finding novel policies that are sufficiently different from existing ones. To encourage the learning policy to consistently converge towards a previously undiscovered local optimum, RSPO switches between extrinsic and intrinsic reward via a trajectory-based novelty measurement during the optimization process. For sufficiently distinct trajectories, RSPO performs standard policy optimization with extrinsic rewards over them, while for trajectories with high likelihood under existing policies, RSPO utilizes an intrinsic diversity reward instead. Experiments show that RSPO is able to discover a wide spectrum of strategies in a variety of domains, ranging from single-agent particle-world tasks and MuJoCo-based continuous control to multi-agent stag-hunt games and Starcraft challenges.

MuJoCo Tasks

Half-Cheetah

Hopper

Walker2d

Humanoid

SMAC

2c_vs_64zg

2m_vs_1z