Learning from Imperfect Demonstrations via Adversarial Confidence Transfer

Abstract

Existing learning from demonstration algorithms usually assume access to expert demonstrations. However, this assumption is limiting in many real-world applications since the collected demonstrations may be suboptimal or even consist of failure cases. We therefore study the problem of learning from imperfect demonstrations by learning a confidence predictor. Specifically, we rely on demonstrations along with their confidence values from a different correspondent environment (source environment) to learn a confidence predictor for the environment we aim to learn a policy in (target environment---where we only have unlabeled demonstrations.) We learn a common latent space through adversarial distribution matching of multi-length partial trajectories to enable the transfer of confidence across source and target environments. The learned confidence reweights the demonstrations to enable learning more from informative demonstrations and discarding the irrelevant ones. Our experiments in three simulated environments and a real robot reaching task demonstrate that our approach learns a policy with the highest expected return. We show the videos of the real robot arm experiments here.

Appendix Link for algorithm details and experimental details here.

Multi-length Partial Trajectory Matching

Feature-Level and Confidence-Level Matching

Experimental Results

We conduct experiments on two MuJoCo environments, a simulated Franka Panda robot arm and a real Franka Panda arm. We compare our method Ours with standard imitation learning GAIL, Dynamics Cycle-Consistency DCC), and variants of our method: Ours w/o Lcon and Ours-Single, where Ours-Single uses a single state-action pair (length-1) matching and feature-level matching, and Ours w/o Lcon adds multi-length partial trajectory matching to Ours-Single, but still uses no confidence-level matching. Finally, we show the results of using the ground-truth confidence to reweight the target demonstrations Oracle.

MuJoCo Environment

Simulated Robot

OT Setting (Learn from Optimal and Out-of-Time Demonstrations, 4 different initializations)

OSC Setting (Learn from Optimal, Suboptimal and Collision Demonstrations, 4 different initializations)

Sim-to-Real Environment

Demonstrations Collection (Success)

Demonstrations Collection (Failure)

Results

Conclusion

We propose an algorithm to learn from imperfect demonstrations, where the demonstrations can be suboptimal or even fail at the task. We learn a confidence predictor by leveraging confidence labels and demonstrations in a different but correspondent source environmen. We show the policy learned by our method outperforms other baselines in various environments.