2025 Competition:
For Human Authors

🗓️ Important Dates

Deadline for submitting questions for the in-person event (UMD mirror: 6/14)

June 1, 2025
- If submitting one packet or more, half the questions are due
June 7, 2025
- All questions due (they will be lightly edited and potentially combined with other submissions)

First, please register to participate in the question writing tournament: https://forms.gle/UopTDcup85x4Zg9M8

Once you start writing questions, share your questions as a Google Doc formatted like this example packet submission and shared with the organizers: ying@umd.edu, yysung53@umd.edu, mgor@umd.edu, nshei0227@gmail.com.
- If you submit via a Google Doc, you're strongly encouraged to either use the interface to check the appropriate difficulty or to have some other mechanism to ensure that they're adversarial.

We have a web interface to help you write questions that specifically stump the computer. We have instructions for how to use the interface. Or you can just start using the interface directly.

If you need "conceptual" guidance on what makes a question difficult for computers to answer, we have a whole page dedicated to tips for writing.

Every packet includes 40 questions at 3.5 dot difficulty (20 tossups and 20 bonus questions).
🛎️ Tossup: aim for questions that a language model can't answer until the last or penultimate line.
🧐 Bonus: aims for 3-part (10 points each) questions that the model gets an average of 15 points.

Submitted question requirements:

Submitted questions should be tossups between 500 and 625 characters at 3-dot difficulty and adhere to quiz bowl style conventions.
Submitted questions must not have previously been shared or submitted elsewhere. Please do not share any information on submitted questions with teams submitting models for the competition.
See this example packet submission for example submitted questions.

Tossups: This approach is outlined in a scientific paper for tossups, but the short version is: models when they hear a tossup output a "confidence" that says how likely they are to know an answer, it is better if this is higher if the models are correct, which corresponds to buzzing. A question is better if humans buzz more often before a machine on a tossup.
Bonuses: For bonuses, we want humans to guess the correct answer more often than computers, detailed at the end of this description of bonus scoring.

We will offer $5 per question that meets our quality guidelines, including tossups and bonus questions
We will offer an additional $50 for any submission set of 10 questions that is better than our staff-written questions. For up to 10 submission sets. If we get more than 10 such submission sets, the prize will be offered to the best ones ranked by the metric.
A single author can win this incentive up to three times.
In addition, we will also offer addition cash prizes for the five best questions ($50) if we have more than five distinct authors submitting qualifying packets.

🧑🏻‍💻 Need any help, or have issues?

Google Sites

Report abuse