Abstract

Robotic reinforcement learning (RL) holds the promise of enabling robots to learn complex behaviors through experience. However, realizing this promise requires not only effective and scalable RL algorithms, but also mechanisms to reduce human burden in terms of defining the task and resetting the environment. In this paper, we study how these challenges can be alleviated with an automated robotic learning framework, in which multi-stage tasks are defined simply by providing videos of a human demonstrator and then learned autonomously by the robot from raw image observations. A central challenge in imitating human videos is the difference in morphology between the human and robot, which typically requires manual correspondence. We instead take an automated approach and perform pixel-level image translation via CycleGAN to convert the human demonstration into a video of a robot, which can then be used to construct a reward function for a model-based RL algorithm. The robot then learns the task one stage at a time, automatically learning how to reset each stage to retry it multiple times without human-provided resets. This makes the learning process largely automatic, from intuitive task specification via a video to automated training with minimal human intervention. We demonstrate that our approach is capable of learning complex tasks, such as operating a coffee machine, directly from raw image observations, requiring only 20 minutes to provide human demonstrations and about 180 minutes of robot interaction with the environment.

In our problem setting, we assume that the task we want the robot to learn is specified by a set of human demonstration videos, each of which is a trajectory of image observations depicting the human performing the task. In contrast to most prior work in robotic imitation learning, we do not assume access to robot demonstrations given through teleoperation or kinesthetic teaching. We also do not assume access to rewards provided through motion capture or other instrumented setups. Our goal is to reduce both the human cost in task specification, by having the robot learn directly from human videos, as well as the human burden during learning.

We devise a robotic learning method which we call automated visual instruction-following with demonstrations (AVID). Starting from a human demonstration, our method translates this demonstration, at the pixel level, into images of the robot performing the task, by means of unpaired image-to-image translation via CycleGAN. In order to handle multi-stage tasks, such as operating a coffee machine, we break up the human demonstration into a discrete set of instruction images, denoting the stages of the task. These instruction images are translated into synthesized robot images, which are then used to provide a reward for model-based RL, enabling the robot to practice the skill to learn its physical execution. This phase is largely automated as the robot learns to reset each stage on its own to practice it multiple times; thus, human supervision is only needed in the form of key presses and a few manual resets. We demonstrate that this approach is capable of solving complex, long-horizon tasks with minimal human involvement, removing most of the human burden associated with instrumenting the task setup, manually resetting the environment, and supervising the learning process.

We evaluate our method and comparisons on two complex and temporally extended tasks: operating a coffee machine and retrieving a cup from a drawer. We compare AVID to learning from full human demonstrations with time contrastive networks (TCN) and an ablation of AVID based on behavioral cloning from observations (BCO). To understand the effects of latent space planning, we also compare to an ablation of AVID based on deep visual foresight (DVF). Finally, we evaluate BCO and behavioral cloning on the same tasks but with access to robot demonstrations, and we also analyze the human supervision burden of AVID.