Diplomacy

The Diplomacy dataset contains pairwise conversations annotated by the sender and the receiver for deception (and conversely truthfulness).   The 17,289 messages are gathered from 12 games.  We train baseline and neural models to predict lies.  More details are available in our ACL 2020 Paper.

What is Diplomacy? 

"Diplomacy plays out on a map of pre-World War I Europe, with each player — it is ideally suited to seven — representing one of the Great Powers of the age: England, France, Germany, Italy, Austria-Hungary, Russia and Turkey. [...] In each of the game’s compulsory negotiation periods, which involve whispering furtively in corners while simultaneously routing eavesdroppers, players in weaker positions band together against those in stronger ones.  What emerges from these sessions, which govern the moves on the board, is a world of quicksilver alliances: joint military campaigns are planned; deals are made, then abrogated, and new agreements arise to take their place. Foe is friend and friend is foe, and it is seldom possible to tell the two apart."

-Allan Calhamer Dies at 81; Invented Diplomacy Game.  (2013, March 6).  New York Times.  

We determine which lies succeed by collecting sender AND receiver annotations.  Below is an example game.

Example Dialog 

Italy: Truly, I don’t want you and England to fight. I am not trying to break you up. I suggested that you take Paris! I want you guy to work together with me against France.

Germany: You don’t want us to fight, yet you betrayed both of our confidence with you in a way that makes us distrust each other?

Italy: I really don’t think that’s a fair description. You guys both wanted to attack each other. I encouraged you both to keep working together.

Germany: Just as long as it suits you. Are you going to give England Mao?

Italy: Hmmm, should I be reading that as angry sarcastic with dagger eyes? (I’m not sure if I’m getting your tone right)

Italy: We’re friends, right? I believe that every single message I’ve sent you all game has been truth, and I’ve gone out of my way to give you candid advice. Are we still friends?

*lies are italicized

Dataset Format and Download   

View Dataset

Each file is a jsonlines file containing an entire game dialog per line. Each object has the following fields: 

• • speaker: the sender of the message

  • recipient: the receiver of the message

  • messages: the raw message string 

  • sender_labels: true or false

  • receiver_labels: true, false, or NOANNOTATION

  • score_delta: the current game score of the sender minus the game score of the recipient

  • absolute_message_index: the index the message is in the entire game, across all dialogs

  • relative_message_index:  the index of the message in the current dialog

  • game_id: which of the 12 games the dialog comes from

Data Format 

Code

Model code, as well as meta game information, can be downloaded at:

https://github.com/DenisPeskov/2020_acl_diplomacy

For any questions, please contact Denis Peskov <dpeskov@cs.umd.edu>.

ACL'20 Paper Bibtex:

@inproceedings{Peskov:Cheng:Elgohary:Barrow:Danescu-Niculescu-Mizil:Boyd-Graber-2020,

Title = {It Takes Two to Lie: One to Lie and One to Listen},

Author = {Denis Peskov and Benny Cheng and Ahmed Elgohary and Joe Barrow    and Cristian Danescu-Niculescu-Mizil and Jordan Boyd-Graber},

Booktitle = {Association for Computational Linguistics},

Year = {2020},

Location = {Seattle},

}