When to Ask for Help: Proactive Interventions in Autonomous RL

Annie Xie, Fahim Tajwar, Archit Sharma*, Chelsea Finn

Abstract: A long-term goal of reinforcement learning is to design agents that can autonomously interact and learn in the world. A critical challenge to such autonomy is the presence of irreversible states which require external assistance to recover from, such as when a robot arm has pushed an object off of a table. While standard agents require constant monitoring to decide when to intervene, we aim to design proactive agents that can request human intervention only when needed. To this end, we propose an algorithm that can efficiently learns to detect and avoid states that are irreversible, and proactively ask for help in case the agent does enter them. On a suite of continuous control environments with unknown irreversible states, we find that our algorithm exhibits both better sample- and intervention-efficiency compared to existing methods.

paint_video.mp4

Proactive Agent Interventions for Autonomous Reinforcement Learning

Overview of our framework PAINT for minimizing human monitoring and supervision when learning in the presence of irreversible states. The agent proactively requests interventions, freeing the human from active monitoring of training. When an intervention is requested, the human resets the environment and provides reversibility labels for the latest experience since the previous intervention.