1st Workshop on Goal Specifications for Reinforcement Learning

Abstract

Reinforcement Learning (RL) agents traditionally rely on hand-designed scalar rewards to learn how to act. The more complex and diverse environments and tasks become, the more difficult it may be to engineer rewards that elicit desired behavior. Designing rewards in multi-agent settings with adversaries or co-operative allies can be even more complicated. Experiment designers often have a goal in mind and then must reverse engineer a reward function that will likely lead to it. This process can be difficult, especially for non-experts, and is susceptible to reward hacking---unexpected and undesired behavior that achieves high reward but does not capture the essence of what the engineer was trying to achieve. Moreover, hand-designed reward functions may be brittle, as slight changes in the environment may yield large, and potentially unsafe, alterations in agent behavior.

The community has addressed these problems through many disparate approaches including reward shaping, intrinsic rewards, hierarchical reinforcement learning, curriculum learning, and transfer learning. Another approach is to avoid designing scalar rewards altogether, and rather focus on designing goals, for example, through inverse reinforcement learning, imitation learning, target images, or multimodal channels such as speech and text.

Each of these approaches are important for obtaining desirable behavior. As such, this workshop will consider all topics related to designing goals for reinforcement learning. The focus will not only be on how to better specify goals in the traditional manner, but also other ways that goals can be defined, and the problems that can be encountered through ill-defined goals.