The standard formulation of Reinforcement Learning lacks a practical way of specifying what are admissible and forbidden behaviors. Most often, practitioners go about the task of behavior specification by manually engineering the reward function, a counter-intuitive process that requires several iterations and is prone to reward hacking by the agent. In this work, we argue that constrained RL, which has almost exclusively been used for safe RL, also has the potential to significantly reduce the amount of work spent for reward specification in applied RL projects. To this end, we propose to specify behavioral preferences in the CMDP framework and to use Lagrangian methods to automatically weigh each of these behavioral constraints. Specifically, we investigate how CMDPs can be adapted to solve goal-based tasks while adhering to several constraints simultaneously. We evaluate this framework on a set of continuous control tasks relevant to the application of Reinforcement Learning for NPC design in video games.

In this first set of experiments, our SAC-Lagrangian agents are trained to solve this navigation task while being subject to only one of the constraints illustrated above. After 3M steps of training, the agent is generally able to solve the task while respecting any of these constraints individually.

Here we demonstrate that when training the agent to respect all of the constraints simultaneously, the task becomes much more challenging.

When training a SAC agent on that task with the main reward function only, the problem becomes simple; the agent rushes to its goal but by doing so it also runs out of energy, ignores the look-at marker, walks into lava, jumps incessantly and uses maximum speed. All of the behavioral requirements are unknown to the agent and are not trivially solved in this environment.

In this work, we argue for the use of Lagrangian methods for behavior specification. This approach allows the RL practitioner to quickly specify behavior using indicator functions instead of having to perform a vast hyper-parameter search over the weights of reward components. We evaluate this framework on the many constraints case in two different environments. Our experiments show that simultaneously satisfying a large number of constraints is difficult and can perpetually prevent the agent from improving on the main task. We first propose to normalize the constraint multipliers, which results in improved stability during training, and suggest to bootstrap the learning on the main objective to avoid getting trapped by the composing constraint set. Our overall method is both easy to implement on top of any existing policy gradient system and can scale across domains with minimal effort from the RL practitioner. Moreover, since the CMDP framework naturally reduces to a regular MDP when no constraint are specified, this framework can be used as a single, by-default method to both constrained and unconstrained problems. We hope that these insights can contribute to a wider use of Constrained RL methods in industrial application projects, and that such adoption can be mutually beneficial to the industrial and research RL communities.