Dr. Marc Lanctot, Google DeepMind

Abstract:

In a previous talk, I summarized progress on development of game-theoretic reinforcement learning in two-player zero-sum games. In this talk, I will give an talk about three new algorithms in this category. The first is Simplex NeuPL, a population-based training method that learns how to respond to any convex mixture of policies it generates during training. The second and third are part of a new class of algorithms that can be broadly categorized as iterative regularization refinement: Regularized Nash Dynamics (R-NaD) and Magnetic Mirror Descent (MMD). Magnetic Mirror Descent adds proximal regularization and leads to linear convergence to quantal-response equilibria (QRE). Both methods simplify convergence to biased equilibria, and then reduce the bias over time, converging arbitrarily close to Nash equilibria. All of these methods avoid the often-necessary problem of explicitly computing the average strategy, but do so in different ways. Finally, I will show how R-NaD has proven to work particularly well at scale, leading to DeepNash: an agent that achieves human-level performance in the game of Stratego.