Joachim Yann Despature1,2, Kazuki Shibata1, Takamitsu Matsubara1
1 Nara Institute of Science and Technology (NAIST), Robot Learning Laboratory
2 École polytechnique fédérale de Lausanne (EPFL)
Vision-Language-Guided Multi-Robot Cooperative Transport
In this study, we address vision-language-guided multi-robot cooperative transport, in which robots cooperatively transport a target object toward a goal landmark specified by a natural language instruction.
Challenge: Perceptual Misalignment
A key challenge in this decentralized setting is perceptual misalignment across robots, where different robots associate the same instruction or environment with different perceived objects or goal regions. Such misalignment undermines coordinated behavior, as inconsistent interpretations directly induce conflicting actions during cooperative transport. This issue is particularly critical in multi-agent systems composed of independent and symmetric agents, each equipped with its own perception and decision-making modules.
Dependent Leader-Follower Design
To mitigate this issue, we depart from the conventional independent and symmetric multi-agent paradigm by adopting a dependent leader–follower design, in which one robot acts as the leader and the other as the follower. By centralizing task-relevant interpretation in the leader, this design mitigates interrobot perceptual misalignment and reduces implementation costs.
Limitations of Symmetric and Independent Learning
Although such a dependent leader-follower structure may appear straightforward, such role differentiation rarely emerges from naive trial-and-error learning. Without explicit inductive biases, independent policy learning tends to converge to symmetric or unstable behaviors rather than a consistent leader-follower structure.
Proposed Method: Consistent Leader-Follower (CoLF)
In this paper, we design and introduce explicit inductive biases to induce a consistent leader–follower structure. Building on this design principle, we propose Consistent Leader–Follower (CoLF), a multi-agent reinforcement learning (MARL) framework for stable leader–follower role differentiation in a two-robot system.
1. Overview of the Proposed Framework
CoLF consists of two key components: (1) an asymmetric policy design that induces leader-follower role differentiation, and (2) a mutual-information-based training objective, which encourages the follower to predict the leader’s action from its local observation. To maximize a variational lower bound of this mutual information, we introduce an auxiliary distribution that enables the follower to predict the leader’s action from its local observation.
2. CoLF Learning Formulation
The leader–follower policy is designed to provide the leader with task-level information while restricting the follower to local observations. Moreover, we introduce Consistency-enhancing term in the training objectives, which encourages the follower to predict the leader’s action from its local observation. The leader and follower policies are jointly optimized under the CTDE framework to balance task execution and consistent cooperative behaviors.
3. VLM-free Training and VLM-based Execution
To reduce computational cost during training while enabling vision-language-guided multi-robot cooperative transport at execution time, we adopt VLM-free training and VLM-based execution. Specifically, the policies are trained using vectorized target object and goal landmark positions, and at execution time they select actions using positions estimated by the VLM.
4. VLM inference
We compute a text-image similarity map using CLIPSeg [1] with RGB images resized to 224 × 224. The similarity map is thresholded at 0.5, and the centroid of the high-similarity region is computed as a 2D image location. This location is back-projected to 3D using depth measurements, a pinhole camera model, and known camera intrinsics and extrinsics.
[1] T. Luddecke and A. Ecker, “Image segmentation using text and image prompts,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 7086–7096.
Simulation
To evaluate the effectiveness of the proposed method, we conducted cooperative transport simulations with two quadruped robots. We considered one-goal-landmark and two-goal-landmark settings to assess robustness to perceptual misalignment caused by viewpoint differences and ambiguous language instructions.
Real Robot Experiment
We validate the proposed approach on real quadruped robots, demonstrating superior performance over the strongest baseline across five cooperative transport scenarios, including challenging cases with perceptual misalignment.
Acknowledgements
This work was supported by the Tateishi Science and Technology Foundation through Research Grant (A).