Introduction

Many autonomous systems such as self-driving cars and industrial robots are complex. In order to deal with this complexity, researchers are increasingly using reinforcement learning (RL) for designing control policies. However, there is one issue that limits the widespread deployment RL in the real system: how autonomous systems can learn robustly without violating the constraints with consideration of satey and other costs throughout exploration of the environment?

Challenges. Learning constraint-satisfying policies is challenging since the policy optimization landscape is no longer smooth. Further, in many cases, the constraints often conflict with the best direction of policy updates to maximize reward. Therefore, we require an algorithm that can make progress in terms of policy improvement without being shackled by the constraints and potentially getting stuck in local minima. A further challenge is that if we do end up with an infeasible (i.e., constraint- violating) policy, we need some efficient means of recovering back to a constraint-satisfying policy.

Contributions. To this end, we develop PCPO – a trust region method that performs policy updates corresponding to reward improvement, followed by projections onto the constraint set. Formally, PCPO, inspired by projected gradient descent, is composed of two steps for each policy update – a reward improvement step and a projection step. We provide theoretical guarantees on performance change for each policy update, and demostrate that our method achieves state-of-the-art performance in items of constraint violation and reward improvement in several benchmarks.

We also provide the code for reproducibility.

paper; slides