Future Events

In Spring 2021, we will be organizing a series of competitions around the following human-computer trivia competitions:

• Adversarial question answering (like our 2018 event)

• Answering questions across cultures

• Answering questions about under-represented groups

• Human-computer collaboration (how well can a team of humans and computers answer questions)

• Detecting wrong facts in statements

Throughout the spring, we'll have small events to culminate in a final event in May 2021.  Watch this space for more details or register for our mailing list to be sure that you don't miss out.

If this isn't of interest to you, we're looking toward the future.  We're looking for:

• Quiz Bowl Tournaments: Who want to use computer support to create questions that are challenging for humans and computers or who are open to computer entrants.  (We also think that this could be used to prevent and better understand cheating.)

• Machine Learning Organizers: Who are looking for an in-person competition to complement, for example, a workshop on machine learning.

• Computer Science Educators: Who want to take part in a machine learning curriculum built around question answering that can lead to an engaging in-person event.

• Corporate Sponsors: Who would like to support our educational programs and competitions.

• Video Editors: Jordan isn't the greatest video editor, and he could put out more videos if he has someone who could create compelling stories from raw footage.

If any of these apply to you, please contact qanta@googlegroups.com.

Contact us and stay informed

Get in-touch

• Please send your questions to qanta@googlegroups.com.

• You can get announcements by joining our human-computer question answering mailing list. We will also post on HSQB.

• Explore other ways for getting involved.

FAQ

Q: Why pyramidal questions?  You're the only idiot in the machine learning / natural language processing community using these questions.

A: primary goal of a dataset/task in machine learning is to distinguish good systems from bad systems.  Single clue datasets (such as TriviaQA or SQuAD) require many more questions to discriminate between top question answerers.  While it's true that quiz bowl questions are longer, it is easier to write a good pyramidal question on a single topic than five good single clue questions on five different topics.

Moreover, because quiz bowl is on a word-by-word basis, it offers far more opportunities to discriminate between question answerers.  Because the questions are pyramidal, the answerer with the deeper knowledge can answer first.

Q: Why have computers and humans play against each other?  Is this a gimmick?

A: Quiz Bowl is designed to be interactive. Humans play against each other in real time (for fun). While machine learning tasks need not be interactive with humans, interruptable questions allow easy comparison against human performance and enable opportunities to teach humans about machine learning, natural language processing, and question answering.

But hopefully it will be fun! 

Q: Computers involved in a trivia tournament ... will the questions suck?

A: The goal of having computers in the loop is to improve the questions from both a scientific perspective and a quiz bowl perspective. Here's how computers can help you write better questions:

• Avoid stock clues in the lead-in

• Automatically find similar questions (to find other interesting clues or avoid repetition)

• Avoid hoses (if the computer thinks the answer might be X, so might a human ... perhaps you can rephrase)

• Automate tasks like pronunciation guides and alternate answer lines (don't know if this will make it in for this iteration, but it's on our todo list!)

Again, we hope that the resulting questions are high quality from a human perspective. We hope that we'll attract good question writers, and that a reasonable reader will recognize them as good questions. We're not going for gimmicks and quirks in terms of the questions.

Q: Should the computer be able to answer all of the questions by the end?

A: No, while your answers have to be in our answer set, if the system on write.qanta.org cannot answer the question, that's fine too!  It probably means that you're doing something unique and interesting. If humans will like and convert the question, then you're doing everything exactly right.

Q: Is this harder than one-line, single sentence QA?

A: From Dwight Wynne:

The pyramidal question is not inherently an easy question. Like any other question, its difficulty is determined by both the answer and clues selected. However, a one-line question is never easier than a pyramidal question containing the same one line. A question will be answered by the union of the sets of [answerers] who recognize and buzz correctly from each clue contained in the question. Since the pyramidal question contains all clues already present in the one-line question, plus additional clues, it must follow that at least as many [answerers] can answer the pyramidal question as the one-line question.

Q: What if we're interested only in complete sentences?  (Or our systems can only answer complete sentences ...)

A: It's possible to only look at individual sentences; i.e., only provide answers after you have a complete sentence.  If you're only interested in single sentences, you can only answer after the first sentence (concrete questions must uniquely identify the answer immediately).  If you can always answer the question after the first sentence, you'll likely do quite well at the overall task.

Thus, quiz bowl is a superset of single sentence QA.  While some questions to require reasoning across sentences, the vast majority of the time it's possible to only answer based on individual, complete sentences in isolation (each sentence in a question getting easier).