NeurIPS 2024 Poster
Yuheng Jing 1,2 Bingyun Liu 1,2 Kai Li 1,2,† Yifan Zang 1,2 Haobo Fu 6 Qiang Fu 6 Junliang Xing 5 Jian Cheng 1,3,4,†
1 Institute of Automation, Chinese Academy of Sciences
2 School of Artificial Intelligence, University of Chinese Academy of Sciences
3 School of Future Technology, University of Chinese Academy of Sciences
4 AiRiA 5 Tsinghua University 6 Tencent AI Lab
Abstract
Opponent modeling is a longstanding research topic aimed at enhancing decision-making by modeling information about opponents in multi-agent environments. However, existing approaches often face challenges such as having difficulty generalizing to unknown opponent policies and conducting unstable performance. To tackle these challenges, we propose a novel approach based on in-context learning and decision-time search named Opponent Modeling with In-context Search (OMIS). OMIS leverages in-context learning-based pretraining to train a Transformer model for decision-making. It consists of three in-context components: an actor learning best responses to opponent policies, an opponent imitator mimicking opponent actions, and a critic estimating state values. When testing in an environment that features unknown non-stationary opponent agents, OMIS uses pretrained in-context components for decision-time search to refine the actor's policy. Theoretically, we prove that under reasonable assumptions, OMIS without search converges in opponent policy recognition and has good generalization properties; with search, OMIS provides improvement guarantees, exhibiting performance stability. Empirically, in competitive, cooperative, and mixed environments, OMIS demonstrates more effective and stable adaptation to opponents than other approaches.
Overview of the Algorithm
On the Left is OMIS's architecture and pretraining procedure: we pretrain a Transformer model for decision-making based on in-context learning. The model consists of three in-context components: an actor, who learns to respond appropriately to the current opponent; an opponent imitator, who mimics the actions of the current opponent; and a critic, who estimates the value of states. We pretrain against various opponent policies to enable each component to acquire the corresponding abilities.
On the Right is OMIS's testing procedure: when facing an opponent with an unknown policy during testing, based on historical context, we guess his current policy by the opponent imitator, guess how to respond to him by the actor, and guess values by the critic, before each real action. We then search a few steps using the actor and opponent imitator, evaluating the imagined returns of each legal action based on rewards during the search and the final search state's value estimated by the critic. By doing this, we determine a search policy that could be more "prescient" than the original actor's policy.
Visualization of OMIS's Search
When playing against opponents with an unseen policy, OMIS’s search promptly evaluates each legal action, predicts the opponent’s actions during the search, and ultimately selects the most advantageous action. There are some interesting phenomena: in Predator Prey, search enables the self-agent (green) to avoid being captured by opponents (red) who use an encirclement policy; in Level-Based Foraging, search allows the self-agent (blue) to cooperate with the opponent (black) to eat apples with a higher level than itself; in OverCooked, search helps prevent the self-agent (green) from blocking the path of collaborators (blue), allowing them to serve dishes smoothly.
Video of OMIS's Gameplay
When playing against opponents with unseen policies, OMIS demonstrates astute gameplay. In Predator Prey, it adeptly evades opponents' pursuits; in Level-Based Foraging, it greedily eats low-level apples and collaborates timely with opponents; in OverCooked, it swiftly adapts to various behaviors from collaborators and serves efficiently.
Predator Prey
Level-Based Foraging
OverCooked
video_oppopi0_ep1.movvideo_oppopi3_ep3.movvideo_oppopi7_ep3.mov