Abstract

Vision–Language–Action (VLA) models have recently demonstrated impressive generalization in robotic manipulation, often attributed to large-scale vision and language pre-training.  However, the action component—responsible for generating concrete motor behaviors—remains fundamentally constrained by limited robot-specific demonstrations. This creates a misleading perception of generalization: while vision and language understanding achieves strong zero-shot capabilities, grounding this understanding to motor controls often fail outside the distribution of observed behaviors.  To systematically study this gap, we introduce \benchmark, a large-scale simulation benchmark for evaluating action generalization in robot learning.  The benchmark comprises x tasks across 19 task categories and two robot morphologies, providing comprehensive coverage of manipulation primitives and long-horizon behaviors. Built on the ManiSkill simulator which enables GPU-parallelized evaluation, ColosseumV2 enables fast and scalable benchmarking, and includes extensive in-domain and out-of-domain evaluation protocols. We provide results for state-of-the-art methods, including Action Chunking Transformers and Pi-family models, establishing strong baselines and highlighting current limitations in motor generalization. By standardizing tasks, metrics, and baselines within a unified leaderboard, ColosseumV2 reduces evaluation overhead and enables reproducible, fair comparisons, accelerating progress toward general-purpose robot policies.

Appendix