Fast Peer Adaptation with Context-aware Exploration

Fast adapting to unknown peers (partners or opponents) with different strategies is a key challenge in multi-agent games. To do so, it is crucial for the agent to efficiently probe and identify the peer’s strategy, as this is the prerequisite for carrying out the best response in adaptation. However, it is difficult to explore the strategies of unknown peers, especially when the games are partially observable and have a long horizon. In this paper, we propose a peer identification reward, which rewards the learning agent based on how well it can identify the behavior pattern of the peer over the historical context, such as the observation over multiple episodes. This reward motivates the agent to learn a context-aware policy for effective exploration and fast adaptation, i.e., to actively seek and collect informative feedback from peers when uncertain about their policies and to exploit the context to perform the best response when confident. We evaluate our method on diverse testbeds that involve competitive (Kuhn Poker), cooperative (PO-Overcooked), or mixed (Predator-Prey-W) games with peer agents. We demonstrate that our method induces more active exploration behavior, achieving faster adaptation and better outcomes than existing methods.

Here we show some visualizations of the PACE agent cooperating with different peer agents in the Overcooked environment. 

As an example, the first peer agent here has a preference for potato onion salad. In episode 1, the PACE agent makes a lettuce broccoli salad, but the peer agent refuses to deliver it; the PACE agent then goes down to the lower room and discovers that the peer agent prefers onion. This active exploration behavior is unnecessary for making a dish but provides valuable information about the peer's preferences. Based on this observation, in episode 2, the PACE agent decides to make a dish containing an onion (tomato onion salad) but still gets rejected. The PACE agent tries again in episode 3 with the potato onion salad and finally succeeds.

For comparison, we also show some visualizations of the Generalist agent, one of our baselines, cooperating with different peer agents in the Overcooked environment.

In the example of potato onion salad, we can see two major differences in behaviors:

1. The Generalist agent never goes down to the lower room, and as such, can only rely on random guesses to explore the peer's preferences. This is due to a lack of exploration rewards.

2. The Generalist agent fails to repeat the successful recipe it has discovered in the earlier episodes. This exploitation failure is potentially due to an inadequate capability of RNNs to summarize long-term contexts.