There is a recent trend of applying multi-agent reinforcement learning (MARL) to train an agent that can cooperate with humans in a zero-shot fashion without using any human data. The typical workflow is to first repeatedly run self-play (SP) to build a policy pool and then train the final adaptive policy against this pool. A crucial limitation of this framework is that every  policy in the pool is optimized w.r.t. the environment reward function, which implicitly assumes that the testing partners of the adaptive policy will be truthfully optimizing the same reward function as well. 

However, humans can be fundamentally biased subject to their own preferences, which can be very different from the environment reward. We propose a more general framework, Hidden-Utility Self-Play (HSP), which explicitly models human biases as hidden reward functions in the self-play objective. By approximating the reward space as linear functions, HSP adopts a simple yet effective technique to generate an augmented policy pool with biased policies. We evaluate HSP  on the Overcooked benchmark. Empirical results show that HSP produces higher rewards than baselines when cooperating with learned human models, manually scripted policies and real humans. The HSP policy is also rated the most assistive policy based on human feedback.

The Overcooked Environment is based on the popular video game Overcooked where multiple players cooperate to finish as many orders as possible within a time limit. Fig.1 demonstrates five layouts we consider, where first three are onion-only layouts. In this simplified version of the original game, two chiefs, each controlled by a player (either human or AI), work in grid-like layouts. Chiefs can move between non-table tiles and interact with table tiles by picking up or placing objects. Ingredients (e.g., onions and tomatoes) and empty dishes can be picked up from the corresponding dispenser tiles and placed on empty table tiles or into the pots. The typical pipeline for completing an order is 

1. players put appropriate ingredients into a pot; 

2. a pot starts cooking automatically once filled and takes a certain amount of time (depending on the recipe) to finish; 

3. a player harvests the cooked soup with an empty dish and deliver it to the serving area.

3.1 Cooperation with Learned Human Models

3.2 Ablation Studies

We investigate the impact of our design choices, including the construction of the final policy pool and the batch size for training the adaptive policy. The results are shown in Fig.1 and Fig.2.

• Policy pool construction:

  • By excluding MEP policies, the HSP variant (HSP w.o. MEP) performs worse in the more complicated layout Counter Circ. while remaining comparable in the other two simpler ones.

  • With policy filtering turned off, even though the policy pool size grows, the performance significantly decays in both Coord. Ring and Counter Circ. layouts, suggests that duplicated biased policies can hurt policy generalization.

• Batch size:

  • In general, we observe that a larger batch size often leads to better training performance. In particular, when the batch size is small, i.e., using 50 or 100 parallel threads, training becomes significantly unstable and even breaks in three layouts. 

3.3 Cooperation with Scripted Policies

We empirically notice that human models learned by imitating the entire human trajectories cannot well capture a wide range of behavior modalities. So, we manually designed a set of script policies to encode some particular human preferences:

• Onion/Tomato Placement, which continuously places onions or tomatoes into the pot.

• Onion/Dish Everywhere, which keeps putting onions or dishes on the counters.

• Tomato/Onion Placement and Delivery, which puts tomatoes/onions into the pot in half of the time and tries to deliver soup in the other half of the time.

Tab. 2 shows the average game reward of all the methods when paired with scripted policies, where HSP significantly outperforms all baselines. In particular, in Distant Tomato, when cooperating with a strong tomato preference policy (Tomato Placement), HSP achieves a 10× higher score than other baselines, suggesting that the tomato-preferred behavior is well captured by HSP.

3.4 Cooperation with Human Participants

We invited 60 volunteers (28.6% female, 71.4% male; median age between 18–30) and divided them into 5 groups for 5 layouts. The experiment has two stages. The first warm-up stage allows participants to play the game freely to explore possible AI behaviors. In the exploitation stage, participants are instructed to achieve a score as high as possible. 

The exploitation stage is designed to test the scoring capability of different AIs. Note that it is possible that a human player simply adapts to the AI strategy when instructed to have high scores. So, in addition to final rewards, we also examine the emergent human-AI behaviors to measure the human-AI cooperation level.

The experiment layouts can be classified into two categories according to whether the layout allows diverse behavior modes. 

• Onion-only Layouts: Fig. 4a shows the average reward in onion-only layouts for different methods when paired with humans. Among these onion-only layouts, all methods have comparable episode reward in simpler ones (Asymm. Adv. and Coord. Ring), while HSP is significantly better in the most complex Counter Circ. layout. Fig. 4b shows the frequency of successful onion passing between the human player and the AI player. The learned HSP policy is able to use the middle counter for passing onions, while the baseline policies are less capable of this strategy.

• Layouts with Both Onions and Tomatoes:

  • Distant Tomato: In Distant Tomato, the optimal strategy is always cooking onion soups, while it is suboptimal to cook tomato soups due to the much more time spent on moving. Interestingly, our human-AI experiments found that humans may have diverse biases over onions and tomatoes. However, all learned baseline policies tend to have a strong bias towards onions and often place onions into a pot with tomatoes in it already. Tab. 3 reports the average number of such Wrong Placements made by different AI players. HSP makes the lowest number of wrong placements and is the only method that can correctly place additional tomatoes into a pot partially filled with tomatoes, labeled Correct Placements. This suggests that HSP is the only effective method to cooperate with biased human strategies, e.g., preferring tomatoes. In addition, as shown in Tab. 3, even when humans play the optimal strategy of cooking onion soups, HSP still achieves comparable performance with other methods.

• Many Orders: In Many Orders, an effective strategy is to utilize all three pots to cook soups. Our experiments found that baseline policies tend to ignore the middle pot. Tab. 4 shows the average number of soups picked up from the middle pot by different AI players. The learned HSP policy is much more active in taking soups from the middle pot, leading to more soup deliveries. Furthermore, HSP achieves a substantially higher episode reward than other methods, as shown in Tab. 4.

4.1 Cooperation with Scripted Policies

Due to the space limit, we only show some representative results, more visualization results could be seen at the link. The files are named as "{blue}-{green}-{reward}.gif" to indicate the roles (two chiefs in blue/green) and the final reward.

Coordination Ring

Dish Everywhere

Counter Circuit

Onion Everywhere

Onion to Middle Counter

Distant Tomato

Tomato Placement

4.2 Cooperation with Human Participants

Coordination Ring

In Coordination Ring, the most annoying thing reported is players blocking each other during movement. To effectively maneuver in the ring-like layout, players must reach a temporary agreement on either going clockwise or counterclockwise. HSP is the only AI able to make way for the other player, while others could not recover by themselves once stuck. For example, both FCP and TrajDiv players tend to take a plate and wait next to the pot immediately after one pot is filled. But they can neither take a detour when blocked on their way to the dish dispenser nor yield their position to the human player trying to pass through.

Counter Circle

In Counter Circuit, one efficient strategy is passing onion via the middle counter. A player at the bottom fetches onions and places them on the counter, while another player at the top picks up the onions and puts them into pots. We find HSP to be the only AI player capable of this strategy in both top and bottom places. Although when HSP is at bottom place, it tends to go top after passing 3 onions, we think that still shows HSP agents are better at adapting to different roles than the baselines.

Distant Tomato

In Distant Tomato, one critical thing is that mixed (onion-tomato) soups give no reward, which means two players need to agree on the soup to cook. All methods perform well when the other player have no preference for tomatoes and focus on onion soups but fail to deal with tomato-preferring partners except for HSP. FCP, MEP, or TrajDiv agents never actively choose to cook tomatoes and may keep putting onions even when a pot has tomatoes in it, resulting in invalid orders. On the contrary, HSP can recognize another player's intention and choose to cooperate. Almost all participants agree that the HSP agent is the best partner to play with in this layout.

Many Orders

In Many Orders, most participants claim that HSP is able to pick up soups from all three pots, while other AIs only concentrate on the pot in front of them and ignore the middle pot even if the human player attempt to use it.