Frequently Asked Questions
What is a shared task?
A shared task is a type of an (informal) research competition organized by a group of researchers or a research organization. The goal of the shared task is to advance the state-of-the-art on a specific research task that is of current interest in the research community.
Teams of researchers from all over the world are invited to participate in the task by developing systems that address the research task. A team consists of one or more members, from one or more collaborating institutes that can be academic or industrial.
The organizers define the research task clearly and provide measures to evaluate the performance of the systems. More importantly, the organizers provide datasets to the participants to help develop their systems. The participating teams can then use the datasets to train (and tune) their systems. At a later stage, a test set will be released on which the participating teams run their systems to get results and submit them to be evaluated by the organizers. The organizers then release the performance scores of the submitted results.
Can participating teams submit a research paper too?
Each participating team that submitted results (called runs) to our Qur'an QA shared task is allowed (and encouraged, but not required) to submit a research paper describing their system and analyzing its performance to WANLP 2023 conference in a special track dedicated to our shared task. The submitted paper will be peer-reviewed, and, if accepted, it will be published within the conference proceedings.
Can a participating team develop multiple systems?
Yes, of course. Our field is experimental, and you probably need to experiment with multiple ideas. So, one team can submit up to 3 runs for the test set per task. All of them will be evaluated by the organizers.
Can a team participate in both tasks?
Yes, of course, but the team has to register in each task separately. Each task is independent from the other and submissions and ranking are separate.
What is expected from the participating teams in a task?
Subscribe to our discussion group to be updated on our shared task.
Download the training and dev set when they are released.
Develop a system that produces results in the given "run file format".
Once the test set is released, run the system on it, get results, submit them on our CodaLab site.
Write a research paper about the developed system and its performance. Submit the paper. This is strongly encouraged, but not required.
If the paper is accepted, you can register at WANLP 2023 and go present your work at the conference in Singapore on the 7th of December 2023.