Motivation

Large-scale supervision has accelerated progress in other fields such as Computer Vision and Natural Language Processing, but policy learning has witnessed no such success.

However, new supervision mechanisms such as RoboTurk allow for 1000s of task demonstrations to be collected in a matter of days. The advent of such mechanisms and large datasets motivates the following question: does a policy learning algorithm necessarily need to interact or can a robust and performant policy be learned purely from external experiences in large datasets?

For example, suppose we are a given a large collection of demonstrations on a pick-and-place task where the robot arm must pick up a soda can and place it in a target region. Given this large dataset, we would like to learn a policy without allowing the policy to collect additional data.

Learning in this setting presents several challenges - the data can consist of suboptimal solution approaches and exhibit substantial diversity. To overcome these issues, we propose Implicit Reinforcement without Interaction at Scale (IRIS), a novel policy learning framework for offline learning from large-scale datasets.

Contributions

• We propose Implicit Reinforcement without Interaction at Scale (IRIS), a policy learning framework that enables offline learning from a large set of diverse and suboptimal demonstrations by selectively imitating local sequences from the dataset.

• We evaluate IRIS across three datasets collected on tasks of varying difficulty. The first dataset is a pedagogical dataset that exhibits significant diversity in the demonstrations. The second dataset exhibits significant suboptimality in the demonstrations and is collected by one user. The third dataset is the RoboTurk dataset collected by humans via crowdsourcing. While our framework can leverage rewards if present in the demonstrations, the experiments only assume sparse task completion rewards that occur at the end of each demonstration.

• Empirically, our experiments demonstrate that IRIS is able to leverage large-scale off-policy task demonstrations that exhibit suboptimality and diversity, and significantly outperforms other imitation learning and batch reinforcement learning baselines.

Why is learning from large-scale demonstrations difficult?

Suboptimality

Demonstrations collected through large-scale supervision can have significant suboptimality.

Diversity

Demonstrations collected through large-scale supervision can consist of diverse task instances and exhibit different solution strategies.

Diversity in Solution Strategies

• Our main insight is to decompose the policy learning problem into two components: (1) low-level goal-conditioned imitation and (2) high-level goal selection.

• The low-level controller is trained to imitate short sequences from the demonstration data. The last observation is treated as the goal. Thus, this controller is able to reach nearby goals.

• The high-level goal selection mechanism consists of a generative model that proposes goals, and a value function that is used to pick the best goals.

• At test-time, given an observation, the high-level selects a new goal, and the low-level is unrolled for T timesteps to try and reach that goal.

• This process repeats until the end of the rollout.

How does IRIS account for suboptimal demonstrations?

• The low-level only needs to reproduce short action sequences, and so it has no need to account for suboptimal actions. The burden of suboptimality is tackled by the high-level.

• The value function is used to choose goals that make significant task progress, accounting for suboptimal demonstrations.

How does IRIS account for diverse demonstrations?

• The goal- conditioned controller is trained to condition on future goal observations at a fine temporal resolution and produce unimodal action sequences.

• Consequently, it is not concerned with modeling diversity, but rather reproduces small action sequences in the dataset to move from one state to another.

• The generative model in the goal selection mechanism proposes potential future observations that are reachable from the current observation - this explicitly models the diversity of solution approaches.

• In this way, IRIS decouples the problem into reproducing specific, unimodal sequences (policy learning) and modeling state trajectories that encapsulate different solution approaches (diversity), allowing for selective imitation.

How does IRIS learn from off-policy data?

• IRIS constrains learning to occur within the distribution of training data.

• The goal-conditioned controller directly imitates sequences from the training data, and the generative goal model is also trained to propose goal observations from the training data.

• The value learning component of the goal selection mechanism mitigates extrapolation error by making sure that the Q-network is only queried on state-action pairs that lie within the training distribution, as in this work.

Datasets and Tasks

Qualitative Results: Learned Policies

Qualitative Results: Interesting Cases

Qualitative Results: Visualizing Goal Selection

Quantitative Results: Task Success Rate

Quantitative Results: Learning Curves

Quantitative Results: Dataset Size Ablation