A shared task is a type of an (informal) research competition organized by a group of researchers or a research organization. The goal of the shared task is to advance the state-of-the-art on a specific research task that is of current interest in the research community.
Teams of researchers from all over the world are invited to participate in the task by developing systems that address the research task. A team consists of one or more members, from one or more collaborating institutes that can be academic or industrial.
The organizers define the research task clearly and provide measures to evaluate the performance of the systems. More importantly, the organizers provide datasets to the participants to help develop their systems. The participating teams can then use the datasets to train (and tune) their systems. At a later stage, a test set will be released on which the participating teams run their systems to get results and submit them to be evaluated by the organizers. The organizers then release the performance scores of the submitted results.
Yes, if you register and request the dataset, you agree to submit a paper. It is not just encouraged, it is a binding commitment you make when receiving the data.
This is strictly prohibited. You agree to register and submit a paper when requesting the dataset. Abandoning the task after receiving data may result in banning participants from future participation.
Yes, of course. Our field is experimental, and you probably need to experiment with multiple ideas. So, one team can submit up to 3 runs for the test set per task. All of them will be evaluated by the organizers.
Yes, of course, but the team has to register in each task separately. Each task is independent from the other and submissions and ranking are separate.
Subscribe to our discussion group to be updated on our shared task.
Download the training and dev set when they are released.
Develop a system that produces expected results.
Once the test set is released, run the system on it, get results, submit them on our CodaLab site.
Write a research paper about the developed system and its performance. Submit the paper.
If the paper is accepted, you can register at EMNLP 2025 and go present your work at the conference in China on the 5th of November 2025.