Write Questions

To make sure the questions are not too easy for computers, we're providing an interface to ensure that writers avoid obvious clues.  You do not have to use the interface to write questions (just submit them), but part of the judging criteria will be whether the questions are appropriately pyramidal for both humans and computers.  However, the answer set is restricted; you should use the interface to select the answers to your questions before you begin writing.

Even if you don’t care about stumping computers, we think it will help you write better questions:

• You’ll be able to avoid stock clues

• You can avoid confusing phrasing that might point to a different answer

• You’ll easily see what other clues other questions have used in the past

We're looking for an independent question editor not connected to Maryland in any way; contact us if you're interested.  If you're interested in getting involved but have limited time, we're also looking for people to judge packets.

If you want to be eligible for prizes, you must register. 

Writing Questions

All questions must be submitted through write.qanta.org through a single account. To make an account, simply login with a new email and password. There can only be one account per email. You can either share passwords or have a single person submit all of the questions.  The goal is to make the first 1-2 lines of the questions unanswerable for computers but otherwise leave the questions something that humans would enjoy (you can see an example of this in this video, also embedded on this page).    

As you write questions, you'll be asked to specify not just the answer but also the category of the questions. The categories are presented at the bottom of this page.  Specifying the answer to the question is important because it allows us to have a fair way to adjudicate whether computers have given the correct answer or not.  Only things that have Wikipedia pages can be answers.  The interface makes it easy to find the right page title, though; just start typing, and it will autocomplete with the possible page titles.

Try not to go over 700 characters for questions, and unless the question is exquisitely crafted, try not to go under 400 characters. The interface will provide the current character count below the question box.

We suggest the following procedure for writing questions.  First, as you’re deciding on answer lines, check the interface to make sure they’re in bounds.  Then, draft the question in whatever editor you’re most comfortable with (e.g., google docs). When you have a first draft, copy and paste it into the interface, edit to make sure the lead-in (at least) isn’t trivially answerable by our baseline system, and then submit.  Inside the interface, certain words are highlighted. These words are the ones that are "important" for our Quizbowl AI system to make its prediction. If you modify those words (e.g., rephrase that sentence) the model will get more confused about the current question.

We also suggest keeping your own backup just in case something goes wrong (we hope it doesn’t, but better safe than sorry). We realize this is slightly more hassle than normal question writing, but this will hopefully lead to better questions and also advance the state of the art in natural language processing.

Rules for Question Writers

• 1. All questions must be submitted at write.qanta.org

  2. All questions should be "clean".  I.e., don't double submit questions that would have gone to another tournament (or will go to another tournament).

  3. Questions should be fit for human consumption: don't write questions that you wouldn't want to listen to.

  4. Questions should be of medium difficulty (HSNCT/PACE NSC level).

  5. All answers must be page titles in Wikipedia in the writing interface (i.e., it's important to use write.qanta.org when selecting answerlines!).

  6. Don't communicate any question information with any team submitting a system to the computer competition.

Complete Rules for Computer Entrants.

Half and full packet requirements

We only ask for tossups only!  No bonuses needed. The distribution is derived from PACE.  Try to get your questions in as early as possible, but half of the questions must be submitted by November 1 (for a full packet, the rest by December 1).  Even if your team is on the waitlist, if you want the benefits of submitting the packet, your packet must be in by the appropriate deadline(s).

Read our preprint for details on round one of the adversarial writing competition.