Short version:

• July 23 Competition at the University of Maryland 

• Using questions written by you that stump computers

• Questions due by July 21, but you need to sign up here. 

• Three ways to compete:

  • Submit questions

  • Answer questions

  • Submit a computer system

Important Dates (all Dates in 2023)

July 20: Questions Due (Double and Half)

July 20: System Submission Deadline

July 22-23: In-person Human vs. Computer Competitions (along with other fun trivia events)

Location

University of Maryland, College Park, MD 

Iribe building: Ground floor/ 2nd floor 

Walkable distance (or take free shuttle 104 https://transportation.umd.edu/spring-exam-week-commencement-modified-parking-shuttle-um-service) from College Park station that has a green line. For parking, You can park at the Visitor Center lot (https://goo.gl/maps/28GeM5iUabC3mNAp6).

Time

July 23 (Sunday): 9am - 12pm

Late Summer TBD: Online Human vs. Computer Competition 

In summer 2023, we are hosting two competitions:

• Writing Questions: Can you write questions that can stump a computer (but that are still answerable by trivia enthusiasts)?

• Human vs. Computer Question Answering: Who is better at answering these questions ... humans or computers?

Prizes are available for computer systems, the best trivia team, and question authors. That will be described below (as well as how to register).  We have more information here that will be useful for those taking part in the competition:

• FAQs about writing Adversarial Questions

• Advice on writing good questions (that are also adversarial)

• Tutorial for Submitting Question Answering Systems

Question Writing Competition

The most important part of this competition is the question writing.  Your questions will be challenged against multiple AI-based QA systems, and your goal is that your questions fool these QA systems but that smart humans can still answer. 

You can participate in this competition by using this application: https://qantanew.web.app/play.  To make sure the questions are not too easy for computers, this interface ensures that writers avoid obvious clues. Alternatively, you may submit your questions as a Google Spreadsheet formatted like this example submission.  Please share this sheet with the organizers as you start writing (ying@umd.edu/yysung53@umd.edu).

Because writing a good question is an art form, we have a separate page detailing what it means to write a good question and how we measure that. 

As you write questions, you'll be asked to specify not just the answer but also the category of the questions. The categories are presented as dropdown menu; choose from the eight categories either in the interface or in the spreadsheet under the "category" column.  

To ensure the questions are high quality, our questions will be inspected and edited by trivia experts.  If your questions look good at first glance, they'll be used as is.  If your questions require minor editing, they will be edited by the trivia experts to fix any stylistic or factual errors.

Question Answering Competition 

After the writing participants have written the questions, we'll have experts edit the questions to make sure they're actually usable and good.

Participating As a Human Player

We will have human competitions on July 22/23! 

Participating As a Computer 

On the other hand, if you think you have a robust QA system that can achieve victory over humans even with these questions, please submit them to the Dynabench submission website (https://dynabench.org/tasks/RQA) using our tutorial as a guide. If neither works for you, just submit your system zip file to yysung53@umd.edu.

The results of the question answering competition will reveal the players who wrote the best questions and the best-performing systems.  We will give out prizes to the best players.

Writing Prizes and Scoring Rubric

We particularly look for questions that falls under below categories (click here to learn how to write good questions). Your total score will be out of 100 points 

1. Discriminability (25 points)

   • Questions that discriminate between top players and bottom players, meaning that your questions should not be too hard or too difficult. This will be calculated by item response theory based on the humans and computers that answer the questions.

2. Difficulty (25 points)

   • Questions that computers struggle with but not as much as humans do are called adversarial questions. We will calculate human accuracy and machine accuracy. Difficulty will count toward your score in two ways:

     1. The questions with the highest difference between human accuracy and computer accuracy will be scored the highest. 

     2.  The human difficulty should be varied (i.e., don't just write difficult questions, the human accuracy should vary).

3. Diversity (25 points)

   • Questions that reflect diverse entities (those that are not typically asked about in trivia questions). We calculate this by taking into account the diversity distribution of countries, locations, and Named Entities. 

4. Quality and Quantity (25 points)

   • We will count the total number of questions that pass the editing process.  The more questions that are used in the competition, the higher your score will be.

We will collectively consider the 4 categories above and integrate them to give prizes to the best writers. We will evaluate each question according to Discriminability, Difficulty, and Diversity. Once the evaluation is done, we will scale the points in each category to give each team a score from 0 to 25. For example, if the submitted questions get 13/25 (Dicriminabiltiy), 20/25 (Difficulty), 10/25 (Diversity) and 24/25 (Quantity) points for each category, the team's final score will be 67. The final score will be out when the competition is over, and will be distributed via email, announced publicly at our events, and posted on this webpage.  

Prizes are given to players/teams who win the writing competition.

• First Place: $500 

• Second place: $250 

• Third place: $150

Please note that the Leaderboard tab does not entirely reflect your score to get prizes, because it only keeps track of the number of questions you wrote, but not the information on how your questions are fooling the machines or discriminating the players. You can refer to it as a partial guidance to see where you stand in the diversity ranking or quantity ranking for bonus points.  

How to Sign Up 

If you want to register as a question writer:

• You can register your team or yourself via this Google Form. 

• After completing the Google form, you will sign up at https://qantanew.web.app/play with the Gmail account you submitted. To make an account, simply login with a new email and password. There can only be one account per email. You can either share passwords or have a single person submit all of the questions. You can check by clicking on the Leaderboard tab and looking for your name after writing a question.  When you come back from doing something else, you do not have to sign up again. 

If you want to register as a human answerer:

• You can register your team or yourself via this Google Form.

If you want to register as a computer answerer:

• You can register your team or yourself via this Google Form.

Submission

If you write your questions in the interface, it will track the categories of your questions and how many questions you wrote for each category.  If you write your questions in a Google Doc, make sure to accurately assign questions to a category.

App Design

• Category tab: You will be able to see how many questions you write for each category on the right upper corner of the page. 

• My Questions tab: You can track down the claims you wrote by selecting the tab on the top left corner of the page. 

• Leaderboard tab: You can compare how many questions you wrote compared to others and check the diversity level of your written questions by selecting the tab on the top left corner of the page. 

Half, Double and Full packet Requirements

 Try to get your questions in as early as possible, you must be submit the questions by July 20 (Half or Full). Your packet must be in by the appropriate deadline(s). The half-packet questions are for those who think they don't have enough time to write a full packet or to gain practice before they submit the full or double-packet questions. 

Organizers and Contact

Please contact us by the email address below if you have any questions or concerns.  

• qanta@googlegroups.com

We gratefully acknowledge the support of Meta for this competition.  However, all claims and assertions are not those of sponsors.