Multi-agent games in dynamic nonlinear settings are challenging due to the time-varying interactions among the agents and the non-stationarity of the Nash equilibria. We consider model-free games, where agent transitions and costs are observed without knowledge of the transition and cost functions. We propose a policy gradient approach to learn distributed policies that follow the communication structure in multi-team games, with multiple agents per team. We model the policies as nonlinear feedback gains, parameterized by self-attention layers to account for the time-varying multi-agent communication topology. 

Distributed Regulation Games

There is an optimality gap between distributed and a centralized optimal method in linear quadratic games (top left). Our method is able to match that gap under network constraints in multi-agent linear quadratic games with unknown cost and dynamics. Given known nonlinear cost and nonlinear multi-agent dynamics, DP-iLQR finds optimal open-loop trajectories for nonlinear multi-agent games that a admit formulation as a potential game (top right). However, we can observe that the differences between the near-optimal performance of our policy and the baseline are minimal after 100 iterations. In terms of collision avoidance and goal-reaching performance, both methods obtain comparable results, despite the differences in the available information. Qualitatively, in a multi-agent nonlinear game, our method (bottom left) is able to recover optimality of base policies (bottom right) computed under known cost and dynamic, despite the knowledge gap.

Multi-Agent Pursuit and Evasion with RL

We study a pursuit-and-evasion game, where two teams compete to optimize their own conflicting costs. We compare our policy with a MLP and an attention-based GNN. To assess the performance, we make the two teams compete against each other using all possible policy combinations, using a nonparametric binomial paired test. As shown in the figures, our method is the best model for evaders and pursuers. We demonstrate a clear performance edge over MLP and GNN, particularly when trained with MAPPO, where the null hypothesis is rejected 9 out of 12 times. This dominance is most evident in the pursuer's role, where we frequently secure win rates exceeding 80\%. While MLP consistently proved to be the weakest baseline across all tests, the comparison with GNN is more nuanced; although we generally lead in on-policy MAPPO settings, GNN remains a robust and highly competitive baseline in off-policy MADDPG settings, although with a significantly higher number of parameters. Interestingly, the defensive capabilities of our method as an evader are more sensitive to opponent strategies. In any case, even in cases where the null hypothesis cannot be rejected, we outperform or tie versus the GNN baseline. Ultimately, the synergy between our attention-based policy parameterization and MAPPO's centralized advantage estimation provides a superior framework for multi-team games.

Experiments

We test our policy parameterization in a real-robot deployment. We zero-shot transfer the policies trained in the pursuit-and-evasion game into the Georgia Tech Robotarium. An important aspect to consider is the presence of control barrier functions that ensure safety during robot operation. These are imposed by the Robotarium platform and cannot be modified by the user or our method. The control barrier functions act on the boundaries of the arena and between robots, preventing collisions. Despite the gap in realism between the gym-based environment used to train the policy and the real-robot setting (including different agent dynamics and safety constraints), the policies display complex pursuit-and-evasion behaviors. For instance, the evaders deceive the pursuers and react in advance to their efforts to catch them, whereas the pursuers counteract and finally corner the evaders. These complex behaviors also happen in real deployments.

Acknowledgments: This work has been supported by ONR N00014-23-1-2353, ONR Global grant N62909-24-1-2081, NSF CCF-2402689 (ExpandAI), in part by Spanish projects PID2024-159284NB-I00, PID2024-159279OB-I00, PID2021-124137OB-I00, by ERDF A way of making Europe and by the European Union NextGenerationEU/PRTR, via project REMAIN S1/1.1/E0111 (Interreg Sudoe Programme, ERDF), by DGA T45-23R, by a Spanish grant FPU19-05700 and by a US-Spain Fulbright grant.