Abstract: Conservative update methods such as Trust Region policy optimization and Proximal policy optimization (PPO) have become the dominant reinforcement learning algorithms because of their ease of implementation and good practical performance. A conventional setup for notoriously difficult queueing network control problems is a Markov decision problem (MDP) that has three features: infinite state space, unbounded costs, and long-run average cost objective. We extend the theoretical framework of these conservative update methods for such MDP problems. The resulting PPO algorithm is tested on a parallel-server system and large-size multiclass queueing networks. The algorithm consistently generates control policies that outperform state-of-art heuristics in literature in a variety of load conditions from light to heavy traffic. These policies are demonstrated to be near-optimal when the optimal policy can be computed. A key to the successes of our PPO algorithm is the use of three variance reduction techniques in estimating the relative value function via sampling. First, we use a discounted relative value function as an approximation of the relative value function. Second, we propose regenerative simulation to estimate the discounted relative value function. Finally, we incorporate the approximating martingale-process method into the regenerative estimator. This is joint work with Mark Gluzman at Cornell.

Bio: Jim Dai is the Leon C. Welch Professor of Engineering at Cornell. He joined Cornell University in 2012 as a professor in the School of Operations Research and Information Engineering (ORIE). Prior joining Cornell, he held the Chandler Family Chair of Industrial and Systems Engineering at Georgia Institute of Technology, where he was a faculty member from 1990 to 2012. He is a Special Term Professor at Tsinghua University. He was a James Riady Distinguished Visiting Professor in Decision Sciences at National University of Singapore (May 2009-Apr 2011), a visiting professor at Aarhus University (Oct 1998-Dec 1998) and Stanford University (Dec 1998-June 1999), and a visiting assistant professor at University of Wisconsin-Madison (Aug 1991-Dec 1991).

Dai is an elected fellow of Institute of Mathematical Statistics and an elected fellow of Institute for Operations Research and the Management Sciences (INFORMS).