Why this Format?

Our format is question answering with the following properties: interruptable, pyramidal, and concrete.

On this page, we describe each of these properties of the questions. We then argue why these are good properties for questions to have from a machine learning standpoint. Because the conclusions apply to both humans and machines, we'll use the generic term "answerer" to describe a person or system answering these questions.

Abstaining / Declaring Answer is Important

Because an answerer can

Dynamic Range

A primary goal of a dataset/task in machine learning is to distinguish good systems from bad systems. Single clue datasets (such as TriviaQA or SQuAD) require many more questions to discriminate between top question answerers. While it's true that quiz bowl questions are longer, it is easier to write a good pyramidal question on a single topic than five good single clue questions on five different topics.

Moreover, because quiz bowl is on a word-by-word basis, it offers far more opportunities to discriminate between question answerers. Because the questions are pyramidal, the answerer with the deeper knowledge can answer first.

Human-Computer Interaction

Quiz Bowl is designed to be interactive. Humans play against each other in real time (for fun). While machine learning tasks need not be interactive with humans, interruptable questions allow easy comparison against human performance and enable opportunities to teach humans about machine learning, natural language processing, and question answering.

FAQ

Below, we address some of the common questions we get about this format.

Q: Is this harder than one-line, single sentence QA?

A: From Dwight Wynne:

The pyramidal question is not inherently an easy question. Like any other question, its difficulty is determined by both the answer and clues selected. However, a one-line question is never easier than a pyramidal question containing the same one line. A question will be answered by the union of the sets of [answerers] who recognize and buzz correctly from each clue contained in the question. Since the pyramidal question contains all clues already present in the one-line question, plus additional clues, it must follow that at least as many [answerers] can answer the pyramidal question as the one-line question.

Q: What if we're interested only in complete sentences?

A: It's possible to only look at individual sentences; i.e., only provide answers after you have a complete sentence. If you're only interested in single sentences, you can only answer after the first sentence (concrete questions must uniquely identify the answer immediately). If you can always answer the question after the first sentence, you'll likely do quite well at the overall task.

Thus, quiz bowl is a superset of single sentence QA. While some questions to require reasoning across sentences, the vast majority of the time it's possible to only answer based on individual, complete sentences in isolation (each sentence in a question getting easier).