Abstract

While reinforcement learning (RL) has become a more popular approach for robotics, designing sufficiently informative reward functions for complex tasks has proven to be extremely difficult due their inability to capture human intent and policy exploitation. Preference based RL algorithms seek to overcome these challenges by directly learning reward functions from human feedback. Unfortunately, prior work either requires an unreasonable number of queries implausible for any human to answer or overly restricts the class of reward functions to guarantee the elicitation of the most informative queries, resulting in models that are insufficiently expressive for realistic robotics tasks. Contrary to most works that focus on query selection to minimize the amount of data required for learning reward functions, we take an opposite approach: expanding the pool of available data by viewing human-in-the-loop RL through the more flexible lens of multi-task learning. Motivated by the success of metalearning, we pre-train preference models on prior task data and quickly adapt them for new tasks using only a handful of queries. Empirically, we reduce the amount of online feedback needed to train manipulation policies in MetaWorld by 20×, and demonstrate the effectiveness of our method on a real Franka Panda Robot. Moreover, this reduction in query-complexity allows us to train robot policies from actual human users.

Policy Visualizations

Note that visualizations are speed up 4x for MetaWorld and 8x for DM Control.

Window Close - 200 Artificial Feedback

Door Close - 200 Artificial Feedback

Door Unlock - 500 Artificial Feedback

Button Press - 500 Artificial Feedback

Drawer Open - 1000 Artificial Feedback

Sweep Into - 2500 Artificial Feedback

Point Mass - 36 Human Feedback

Reacher - 48 Human Feedback

Window Close - 64 Human Feedback

Door Close - 100 Human Feedback

Notice that in the door close environment in particular, very different strategies are learned. Our method (few-shot) learns to move to the door handle and then close the door from it. On the other hand, PEBBLE tends to learn to slam into the corner of the door, and cause the door to close. Our method recovers a policy more similar to what is desired by the human user as seen below.

Panda Reach - 200 Artificial Feedback

Panda Block Push - 2000 Artificial Feedback

User Queries

Here we show selected queries for both our method and PEBBLE on each of the environments used for human training. The top behavior was preferred by the human user. In the downloadable content we include the full images shown to users during policy training.