LAN2CB

From Language to Collective Behavior: Compositional Multi-Robot Coordination Using Large Language Models

Abstract

Multi-robot coordination has traditionally relied on a task-specific and expert-driven pipeline, where natural language mission descriptions are manually translated by domain experts into mathematical formulation, algorithm design, and executable code. This conventional process is labor-intensive, inaccessible to non-experts, and inflexible to changes in mission requirements. Here, we propose LAN2CB (Language to Collective Behavior), a novel framework that leverages large language models (LLMs) to streamline and generalize the multi-robot coordination pipeline. LAN2CB directly converts natural language mission descriptions into executable Python code for multi-robot systems through two key components: (1) Mission Decomposition for Task Representation, which parses the mission into a task graph with dependencies, and (2) Code Generation, which uses the task graph and a structured knowledge base to generate deployable robot control code. We further introduce a dataset of natural language mission specifications to support development and benchmarking. Experimental results in both simulation and real-world settings show that LAN2CB enables effective and flexible multi-robot coordination from natural language, significantly reducing the need for manual engineering while supporting generalization across mission types. Website: https://sites.google.com/view/lan2cb

Overview

A comprehensive workflow demonstration

Simulation Results

Mission #1

This mission includes three tasks.

[1] The left three robots visit the center of region 0, region 1, and region 2 separately.

[2] The right three robots position themselves around region 3.

[3] All robots form a circle around region 3.

Task dependency

[1]

[2]

([1], [2]) -> [3]

Mission #2

This mission includes three tasks.

[1] All robots herd three dynamic objects into region 1.

[2] Three robots block the left side of region 1.

[3] Two robots block the right side of region 1.

Task dependency

[1]

[1]-> [2]

[1] -> [3]

Mission #3

This mission includes three tasks.

[1] The left four robots herd left two dynamic objects into region 0.

[2] The right four robots herd right two dynamic objects into region 0.

[3] All robots enclose region 0 by forming a circle shape around region 0.

Task dependency

[1]

[2]

([1], [2]) -> [3]

Mission #4

This mission includes two tasks.

[1] All robots form 'H'.

[2] All robots from 'I'.

Task dependency

[1]

[1] -> [2]

Mission #5

This mission includes two tasks.

[1] The left two robots visit points [-5.0, -4.0] and [-3.0, -4.0] separately by following zigzag paths.

[2] The right two robots visit points [-1.0, -4.0] and [1.0, -4.0] separately by following spiral paths.

Task dependency

[1]

[2]

Mission #6

This mission includes six tasks.

[1] The left two robots follow the left purple dynamic object for 10 seconds.

[2] The middle two robots follow the middle purple dynamic object for 10 seconds.

[3] The right two robots follow the right purple dynamic object for 10 seconds.

[4] The left two robots enter the region 0 from its left edge.

[5] The middle two robots enter the region 0 from its top edge.

[6] The right two robots enter the region 0 from its right edge.

Task dependency

[1]

[2]

[3]

[1] -> [4]

[2] -> [5]

[3] -> [6]

Mission #7

This mission includes two tasks. One task has a trigger.

[1] The left three robots need to visit [-1.0, 5.0], [1.0, 5.0], and [0.0, 6.0] separately. If any of these three robots move 1.5 units close to the forbidden region, the trigger activated, and these three robots need to stop the current task and immediately go to the region 0. The point visitation task will take over by the other three robots after they finish their own task.

[2] The right three robots need to enclose the region 1 by forming a circle shape.

Task dependency (Do not include trigger event initially)

[1]

[2]

After trigger activated, the task dependency would be updated.

[3] The left three robots need to visit region 0.

[4] The right three robots need to enclose the region 1 by forming a circle shape.

[5] The right three robots need to visit [-1.0, 5.0], [1.0, 5.0], and [0.0, 6.0] separately.

[3]

[4]

[4] -> [5]

Mission #8

This mission includes two tasks. The mission has a trigger.

The trigger is: If the center of the left two purple dynamic objects and the center of the right two purple dynamic objects are smaller than 3.0 units, all robots would regroup to one group and herd all purple dynamic objects to region 2.

[1] The left four robots need to herd left two purple dynamic objects to region 0.

[2] The right four robots need to herd right two purple dynamic objects to region 1.

Task dependency (Do not include trigger event initially)

[1]

[2]

After trigger activated, the task dependency would be updated.

[3] All robots herd all purple dynamic objects to region 2.

Mission #9

This mission do not specify the task explicitly.

The mission is all robots need to herd all purple dynamic objects and make the purple dynamic objects cover 60% of the area of region 0.

After prompt the LLM (GPT4.1), the LLM split the mission to four tasks.

[1] All robots herd all dynamic objects to the top left subregion of the region 0.

[2] All robots herd all dynamic objects to the top right subregion of the region 0.

[3] All robots herd all dynamic objects to the bottom left subregion of the region 0.

[4] All robots herd all dynamic objects to the bottom right subregion of the region 0.

Task dependency

[1]

[1] -> [2]

[2] -> [3]

[3]-> [4]

Real World Demonstrations

Mission

This mission includes three tasks.

[1] The top two robots each visit the top-left and bottom-right points of the purple region, respectively.

[2] The bottom two robots each visit the top-left and bottom-right points of the dark yellow region, respectively.

[3] All robots form a reverse 'C' shape around the left red region.

Task dependency

[1]

[2]

([1], [2]) -> [3]

Page updated

Google Sites

Report abuse