Adversarial Writing of Quizbowl Questions

Website to accompany the TACL 2019 paper "Trick Me If You Can: Human-in-the-loop Generation of Adversarial Question Answering Examples". Presented at ACL, Poster Session 3 (Arsenale: July 26, Monday 16:00)


Visit the data section of QANTA to download:

  • The adversarial datasets

  • Edit histories of the questions to see how the authors got to the adversarial questions

  • Additional Quizbowl data

For the impatient, there are human readable versions of the prelim and final questions used in the Dec 15 event.

Live Competition

Videos, questions, and winners on the description of the Dec 15, 2018 event.


Visit to write adversarial Quizbowl questions.


Visit our Github repository to see the code for the interface, models, and interpretations.

Our data is easily used in python:

import json

with open('qanta.tacl-trick.json') as f:

data = json.load(f)

print(f"Bibtex: {data['bibtex']}")

print(f"Version: {data['version']}")

print('Printing first few questions')


for q in data['questions'][:3]:



# 'answer' is the unnormalized answer, 'page' is the true answer

print('Page: ', q['page'])

# How the question was authored

print('Interface:', q['interface'])


A playlist of all our videos explaining the adversarial question writing process, examples of questions, and how a live competition between computers and humans played out.

Acknowledgement: The submission of computer systems was made possible by an AWS research credit award from Amazon.

Eric Wallace

Pedro Rodriguez

Shi Feng

Ikuya Yamada

Jordan Boyd-Graber