Safe Deep RL in 3D Environments
using Human Feedback

Note: if videos below don't load, try a different browser. Firefox and Safari seem to be more reliable for these videos than Chrome.

Agent evaluation episodes

Below we show videos of our ReQueST agent performing the task. The left panel shows observations from the real environment, along with a procedural reward function that gives a reward of +1 for each apple eaten. The right panel shows observations from plans produced by model predictive control along with rewards predicted by the reward model.

These examples are from the cliff edge environment, in the large arena size.

Cherry-picked examples

Failure mode 1: reward bonus hacking

The learned reward function predicts rewards based on the distance to the nearest apple. To encourage the agent to actually eat the apples by moving through them rather than just moving up close to an apple, we give a reward bonus if the predicted reward rapidly drops. However, the agent is sometimes also able to trigger the bonus by moving close to an apple and then spinning in-place.

Failure mode 2: poor reward model quality

Various types of failure can also result from the reward model not being sufficiently accurate.

In the example on the left, the reward model gives an output of only 0.5 for being close to an apple - not high enough to trigger the reward bonus that would cause the agent to actually move through the apple and eat it.

In the example on the right, the reward model output reaches a maximum of 0.4 while still some distance from the nearest apple. The cause of this failure is ambiguous. Possibilities include a) optimisation failure (during MPC, no rollouts were selected that moves towards the apple); b) a local maximum (the reward model would output a high reward for being much closer to an apple, but the output would drop when between the current position and that closer position); or c) reward model quality (the reward model outputs a maximum of 0.4 no matter how close the agent moves to the apple). By examining MPC rollout samples and/or taking interactive control of the agent to see how the reward model output changes with distance, we can determine that in this case the failure was caused by c).

Model-free agent

Below we show the failure of the model-free baseline agent in the large dangerous blocks environment. instead of collecting apples, the agent moves straight to the blocks, giving itself high reward by pushing the blocks away from each other.

Reward sketching interface

Humans provide the feedback used to train the reward model using reward sketches. Below we show the interface used to provide these sketches being using to sketch the rewards for a single trajectory.

Trajectories optimised for human feedback

To generate interesting trajectories for the human to provide feedback on, we can optimise trajectories for various criteria. For example, we can optimise trajectories for maximum reward as predicted by the reward model, exposing false-positives: cases where the reward model outputs a higher reward than it should. Similarly, by optimising trajectories for minimum predicted reward, we can find false-negatives.

Example trajectories below are chosen randomly - not cherry-picked.

Maximum predicted reward, cliff edge environment

Maximum predicted reward, dangerous blocks environment

Minimum predicted reward, cliff edge environment

Minimum predicted reward, dangerous blocks environment