RoboCLIP:

Reward specification is a notoriously difficult problem in reinforcement learning, requiring extensive expert supervision to design robust reward functions. Imitation learning (IL) methods attempt to circumvent these problems by utilizing expert demonstrations instead of using an extrinsic reward function but typically require a large number of in-domain expert demonstrations. Inspired by advances in the field of Video-and-Language Models (VLMs), we present RoboCLIP, an online imitation learning method that uses a single demonstration (overcoming the large data requirement) in the form of a video demonstration or a textual description of the task to generate rewards without manual reward function design. Additionally, RoboCLIP can also utilize out-of-domain demonstrations, like videos of humans solving the task for reward generation, circumventing the need to have the same demonstration and deployment domains. RoboCLIP utilizes pretrained VLMs without any finetuning for reward generation. Reinforcement learning agents trained with RoboCLIP rewards demonstrate 2-3 times higher zero-shot performance than competing imitation learning methods on downstream robot manipulation tasks, doing so using only one video/text demonstration. 

RoboCLIP Overview. A Pretrained Video-and-Language Model is used to generate rewards via the similarity score between the encoding of an episode of interaction of an agent in its environment, zv with the encoding of a task specifier zd such as a textual description of the task or a video demonstrating a successful trajectory. The similarity score between the latent vectors is provided as reward to the agent as a sparse reward at the end of the episode.

Button-Press: Results on Using Video and Language Demonstrations in the Button Press. The above gifs visualize the zero-shot policy obtained by training on RoboCLIP rewards. 

Door-Close: Results on Using Video and Language Demonstrations in Door Close. The Language conditioned policy moves towards the door but fails to finish the task. The video conditioned policy completes the task successfully zero-shot.  

Drawer-Close: Results on Using Video and Language Demonstrations in Drawer Close. Both the video and language conditioned policies complete the task successfully zero-shot.  

Button-Press Human: The rewards generated from human videos are somewhat noisier. But the robot moves towards the correct object and suffers from perceptual errors missing the red button while moving close to it. This policy is subsequently fine-tuned using a single demonstration below. 

Drawer-Open Human: The rewards generated from human videos are somewhat noisier. The robot moves grasps the correct object but fails to fully open it. 

Door-Open Human: The rewards generated from human videos are somewhat noisier. Here too, the robot moves grasps the handle to open the door but fails to fully open it.