(Accepted by CoRL25)
Ruize Zhang, Sirui Xiang, Zelai Xu, Feng Gao, Shilong Ji, Wenhao Tang,
Wenbo Ding, Chao Yu† , Yu Wang†
† Corresponding Authors
Tsinghua University
In this paper, we tackle the problem of learning to play 3v3 multi-drone volleyball, a new embodied competitive task requiring high-level strategic coordination and low-level agile control. The task is turn-based, multi-agent, and physically grounded, posing significant challenges due to its long-horizon dependencies, tight inter-agent coupling, and the underactuated dynamics of quadrotors.
To address this, we propose Hierarchical Co-Self-Play (HCSP), a hierarchical reinforcement learning framework separating centralized strategic decision-making from decentralized low-level motion control. We design a three-stage population-based training pipeline to enable both strategy and skill to emerge from scratch without expert demonstrations: (I) training diverse low-level skills, (II) learning high-level strategy via self-play with fixed low-level controllers, and (III) joint fine-tuning through co-self-play.
Experiments show that HCSP achieves superior performance, outperforming non-hierarchical self-play and rule-based hierarchical baselines with an average 82.9% win rate and a 71.5% win rate against the two-stage variant. Moreover, co-self-play leads to emergent team behaviors such as role switching and coordinated formations, demonstrating the effectiveness of our hierarchical design and training scheme.
Fig. 1 VolleyBots' 3v3 Competitive Task
Our work aims to tackle the 3v3 Multi-Drone Volleyball Task proposed in VolleyBots. VolleyBots introduces a drone volleyball testbed built on the OmniDrones simulator, leveraging NVIDIA Isaac Sim's high-fidelity physics engine for efficient training. Our work focuses on VolleyBots' 3v3 competitive task, which slightly simplifies real-world volleyball rules for drone implementation.
Two teams of three drones each compete on a scaled-down 6m × 12m court while maintaining the official net height of 2.43m. Each team must return the ball across the net within three consecutive hits using coordinated actions. A team loses points for failing to return the ball, hitting out-of-bounds, violating turn-based interaction rules, or having drones collide with the ground. Each episode lasts a maximum of 15 seconds, starting with a randomized serving team. The episode ends when either one team scores or the time limit is reached. This competitive task presents dual challenges of precise motion control and coordinated team strategy.
Fig. 2 Overview of HCSP architecture: an event-driven high-level strategy handles strategic decisions, while multiple low-level skills manage continuous control. Each drone (i) runs its low-level skill at 50Hz, taking local observation (o_i^L) and high-level tactical parameters to produce continuous action (a_i^L). The high-level strategy activates only on discrete events (racket strike or ball crossing the net), observes (o^H), and outputs (a^H) to choose each drone’s skill and supply tactical information. It is implemented as a shared MLP with three output heads, one per drone.
The overview of the Hierarchical Co-Self-Play (HCSP) Reinforcement Learning architecture is shown in Fig. 1, while the main contributions of this work are summarized as follows:
We conducted the first systematic study of the 3v3 multi-drone volleyball task, which combines strategic team play, precise motion control, and high-speed aerial agility in a turn-based, multi-agent physical environment.
We propose Hierarchical Co-Self-Play (HCSP), a hierarchical reinforcement learning method that integrates a centralized high-level strategy for team tactics and decentralized low-level skills for agile control.
We introduce a three-stage population-based training pipeline that enables the emergence of both motion skills and tactical strategies without expert demonstrations. The final co-self-play stage fosters emergent behaviors such as role switching and formation specialization.
In extensive experiments, HCSP outperforms both non-hierarchical self-play and rule-based hierarchical baselines, achieving an 82.9% average win rate against the baselines and a 71.5% win rate against the two-stage variant.
To address the challenges of precise motion control and coordinated team strategy, we propose Hierarchical Co-Self-Play (HCSP), a hierarchical reinforcement learning (HRL) method that decomposes the overall policy into an event-driven high-level strategy for team tactics and low-level skills for individual maneuvers. HCSP's training pipeline comprises three stages:
We train each low-level skill via RL using policy chaining, which addresses the challenge of executing temporally adjacent primitives smoothly by incrementally training each new skill in the context of previously learned ones, resulting in a robust pool of motion primitives.
With the low-level skill pool frozen, we pretrain the high-level strategy through population-based team-level self-play, enabling it to acquire initial cooperative and competitive tactics.
We utilize co-self-play to co-optimize both skill levels through team-level self-play, where the high-level strategy dynamically selects from continuously improving low-level skills while simultaneously refining their execution.
Table 1 Description of seven low-level skills acquired in stage I & corresponding tactical observations provided by high-level strategy.
Low-level skills focus on controlling individual drones to execute specific motion primitives without tactical reasoning. Each drone is regarded as a low-level agent.
Due to the challenge of ensuring smooth transitions between temporally adjacent low-level primitives, especially when control switches across drones or skill types, we employ policy chaining to iteratively train each low-level skill for seamless execution under the event-driven high-level strategy. For example, under policy chaining, the Attack skill is trained following a teammate’s execution of the Set skill, enabling coordinated multi-agent behaviors. This approach enhances intra-drone motion continuity and inter-drone cooperation. Our experiments show that policy chaining significantly improves the reliability of sequential skill execution and accelerates learning.
Fig. 3 Training Schedule for Stage II.
Fig.4 Sample reallocation for high-level strategy training.
High-level strategies enable team-level decision-making, with each team modeled as a centralized high-level agent.
In this stage, we freeze the low-level skills and pretrain the high-level strategy through the team-level self-play method. We employ an iterative self-play procedure combining population-based evaluation with Nash equilibrium (NE)-based opponent sampling. When the candidate’s win rate exceeds a threshold or a maximum step count is reached, it is added to the shared population. We then perform pairwise evaluations of all population policies to produce a new win-rate matrix, update the NE-based sampling distribution accordingly, and repeat the cycle.
Since the high-level strategy is event-driven, its action steps are temporally sparse. To address this, we apply sample reallocation (Fig. 4), which extracts transitions at event-triggered timesteps and attributes cumulative rewards over the intervals between events
Fig. 5 llustration of the co-self-play stage (Stage III).
As high-level strategies can motivate low-level skill development and, in turn, improvements in low-level skills inform high-level decision-making, co-self-play is essential to iteratively optimize both layers.
In Stage III, we jointly refine both high- and low-level policies (Fig. 5). Similar to Stage II, we train only one team at a time against a frozen opponent, with the candidate high- and low-level skills learning together while the opponent’s policies remain frozen. Whenever the candidate’s win rate exceeds a threshold, the newly trained low-level policies are added to the low-level pool. As the low-level skill pool has grown, the high-level actor’s output layer needs to accommodate the expanded action set. We then freeze the augmented low-level pool and train a new high-level strategy from scratch against the former high-level strategy and the original low-level skill pool.
Fig. 6 HCSP performance against baseline policies.
Fig.6 shows the win rates when HCSP plays against each of the five baseline methods. HCSP achieves an average win rate of 82.9% across all matchups, demonstrating its strong competitive advantage. Notably, it significantly outperforms all four non-hierarchical game-theoretic methods. Moreover, HCSP attains a 71% win rate against the strongest baseline, the rule-based hierarchical Bot policy provided in VolleyBots. These results indicate that HCSP both outperforms traditional self-play strategies and competes strongly against the handcrafted hierarchical policy.
In this section, we evaluate the impact of Stage III co-self-play within the full training pipeline by comparing the final HCSP policy (i.e., the Stage III policy) with its counterpart trained using only the first two stages (i.e., the Stage II policy), through both quantitative and qualitative analyses.
Table 2 Win rates of Stage II policy and Stage III policy against different opponents.
To evaluate the effectiveness of Stage III training, we first conduct a head-to-head tournament between the Stage II and Stage III policies, as shown in the first two columns of Tab. 2. The Stage III policy achieves a 71.5% win rate over the Stage II policy, demonstrating clear improvement under the co-self-play framework.
To assess absolute robustness, we construct a Nash-averaging evaluator. As shown in Tab. 2, the Stage III policy improves its win rate from 31.4% (Stage II) to 47.6%, a 16.2% absolute gain. Notably, the Stage III policy reaches approximately 50% win rate against the Nash-averaged evaluator, indicating competitive performance near the Nash Equilibrium of the sampled strategy set.
Fig. 7 Cross-play win rate heatmap among ten randomly picked policies.
Fig. 8 Sequence of six temporally sampled frames illustrating an emergent team behavior. (a) The opponent team passes the ball to the setter. (b) The opponent team sets the ball to the attacker. (c) The opponent team attacks the ball towards our side. (d) Our team passes the ball to the setter. (e) Our team performs a dump shot, sending the ball directly onto the opponent’s court. (f) The opponent team fails to return the ball, so our team scores the point.
From a qualitative perspective, the experimental results further show that Stage III's co-self-play induces emergent low-level behaviors that were not designed in the initial stage. In Stage I, policy chaining yields three motion primitives, Pass, Set, and Attack, with high accuracy, so each team always uses three consecutive contacts (Pass to the setter drone, Set to the attacker drone, Attack over the net). During Stage II, with low-level skills frozen, the high-level strategy continues to learns this three-touch sequence. In contrast, after Stage III, we observe a novel two-touch maneuver, akin to the real-world volleyball dump shot, in which the setter itself redirects the ball over the net instead of raising it for a teammate.
This emergent dump skill requires no additional reward design and illustrates two key effects of co-self-play: low-level primitives can evolve beyond their initial definitions, and the high-level strategy adapts its skill allocation (for example, omitting Attack and assigning Hover to the would-be attacker drone) to leverage these new behaviors. Fig. 8 presents a case study in which one team uses the emergent dump skill to score a point.
Fig. 9 Ablation study on policy chaining in Stage I: policy chaining enables successful transition from Attack to Hover, whereas single-policy fails.
In Stage I, we employ policy chaining, an iterative training method designed to ensure smooth transitions between temporally adjacent low-level skills. We compare policy chaining with a single-policy approach by training the "Attacking the ball to the opponent's side, followed by Hovering to maintain stability" behavior under the same reward settings. As shown in Fig.9, both approaches achieve similar performance in the Attack phase. However, only policy chaining successfully learns to transition to Hover after Attack. This demonstrates that policy chaining reduces exploration difficulty by decomposing the complex behavior into sequential, specialized policies.
Fig. 10: Policy population evolution win-rate heatmap in Stage II. Left: without sample reallocation. Right: with sample reallocation.
We compare standard minibatch sampling to our sample reallocation method by visualizing the high-level strategy’s evolution over five self-play iterations using win-rate heatmaps (Fig. 10). In these heatmaps, off‑diagonal entries reflect each new policy’s win rate against its predecessors. Using sample reallocation, these off‑diagonal cells become strongly favorable much earlier, indicating faster skill progression, and exhibit lower win-rate variance, indicating more stable training.
Also, we compare two policy populations in 500 head‑to‑head matches averaged over three random seeds, where each side samples a strategy from its population according to its NE–based distribution at the start of each match. The sample reallocation population wins 90.5% ± 4.5% of matches against the standard‑sampling population.