Abstract

One difficulty in using artificial agents for human-assistive applications lies in the challenge of accurately assisting with a person’s goal(s). Existing methods tend to rely on inferring the human’s goal, which is challenging when there are many potential goals or when the set of candidate goals is difficult to identify. We propose a new paradigm for assistance by instead increasing the human’s ability to control their environment, and formalize this approach by augmenting reinforcement learning with human empowerment. This task-agnostic objective increases the person’s autonomy and ability to achieve any eventual state. We test our approach against assistance based on goal inference, highlighting scenarios where our method overcomes failure modes stemming from goal ambiguity or misspecification. As existing methods for estimating empowerment in continuous domains are computationally hard, precluding its use in real time learned assistance, we also propose an efficient empowerment-inspired proxy metric. Using this, we are able to successfully demonstrate our method in a shared autonomy user study for a challenging simulated teleoperation task with human-in-the-loop training.

Key Insight

Agents can assist humans without inferring their goals or limiting their autonomy by instead increasing the human’s controllability of their environment – in other words, their ability to affect the environment through actions. We capture this via the information theoretic quantity of empowerment. In our method, Assistance via Empowerment (AvE), we formalize the learning of assistive agents as an augmentation of reinforcement learning with a measure of human empowerment. The intuition: by prioritizing agent actions that increase the human’s empowerment, we enable the human to more easily reach whichever goal they want, even though the agent has no information about their goal.

In the figure above, the left image shows a robot mistakenly inferring that the human wants door B. In the right image, the robot assesses that the human’s empowerment would increase with doors B and C open. Naively opening all doors will not increase empowerment as A is too small for the person and D leads to the same location as C.

Shared Workspace: Comparison with Goal Inference | Code

Training an agent to provide assistance is challenging when the human’s goal is unknown because that makes it unclear what the agent should do. Approaches to assistance in both shared workspace and shared autonomy settings have focused on inferring the human’s goal (or, more broadly, the hidden reward parameters) from their ongoing actions, building on tools from Bayesian inference and Inverse Reinforcement Learning. However, goal inference can fail when the human model is misspecified, e.g. because people are not acting noisy-rationally, or because the set of candidate goals the agent is considering is incorrect. In such cases, the agent can infer an incorrect goal, causing its assistance (along with the human’s success) to fail. To highlight situations where AvE overcomes challenges of goal inference, we simulate the following shared workspace scenario.

We simulate a person (blue circle) attempting to navigate to a goal (yellow star). The grid also contains blocks (grey squares) that the human cannot move, but an assistive agent can move them. However, the assistant does not know where the human goal is. We test two strategies: goal inference and AvE. Our results show that in the ideal case where the goal set is precise and accurate, the agent provides the most efficient assistance. However, in cases where the goal set is misspecified or too large, the agent can incorrectly infer the goal and accidentally block the human's true goal, leading to a failure mode. AvE avoids this failure mode since blocking the human would reduce their empowerment, and ultimately consistently provides assistance without knowledge of the goal. In the figure below, sample rollouts are shown for each assistant type. We find across a variety of gridworld initializations that AvE consistently achieves 100% success, albiet at a slight performance cost compared to the ideal goal inference method. At the same time, the unideal goal inference methods achieve lower success rates due to the aforementioned failure modes.

BibTex