Minttu Alakuijala¹ Reginald McLean² Isaac Woungang² Nariman Farsad² Samuel Kaski¹˒³ Pekka Marttinen¹ Kai Yuan⁴
1 Department of Computer Science, Aalto University
2 Department of Computer Science, Toronto Metropolitan University
3 Department of Computer Science, University of Manchester
4 Intel Corporation
TMLR, February 2025
Paper
Code
Natural language is often the easiest and most convenient modality for humans to specify tasks for robots. However, learning to ground language to behavior typically requires impractical amounts of diverse, language-annotated demonstrations collected on each target robot. In this work, we aim to separate the problem of what to accomplish from how to accomplish it, as the former can benefit from substantial amounts of external observation-only data, and only the latter depends on a specific robot embodiment. To this end, we propose Video-Language Critic, a reward model that can be trained on readily available cross-embodiment data using contrastive learning and a temporal ranking objective, and use it to score behavior traces from a separate actor. When trained on Open X-Embodiment data, our reward model enables 2x more sample-efficient policy training on Meta-World tasks than a sparse reward only, despite a significant domain gap. Using in-domain data but in a challenging task generalization setting on Meta-World, we further demonstrate more sample-efficient training than is possible with prior language-conditioned reward models that are either trained with binary classification, use static images, or do not leverage the temporal information present in video data.
Overview
Overview of Video-Language Critic: Our similarity function, Sθ , is trained using video-caption pairs (vⁱ₁:ᴛ , cⁱ). The visual encoder (ViT) is applied separately to each video frame vₜ to produce a sequence of image features, which are appended to the caption embedding produced by the text encoder. The temporal aggregator then predicts a similarity score for each time step t of the video. We use cross-entropy and sequence ranking objectives to encourage the predicted scores to be high for matching video-caption pairs and low for mismatching pairs, and to monotonously increase over a successful execution.
Video Presentation
@article{alakuijala2025video,
title={Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics},
author={Alakuijala, Minttu and McLean, Reginald and Woungang, Isaac and Farsad, Nariman and Kaski, Samuel and Marttinen, Pekka and Yuan, Kai},
journal={Transactions on Machine Learning Research (TMLR)},
year={2025},
}