Frequently Asked Questions
What is a shared task?
A shared task is a type of an (informal) research competition organized by a group of researchers or a research organization. The goal of the shared task is to advance the state-of-the-art on a specific research task that is of current interest in the research community.
Teams of researchers from all over the world are invited to participate in the task by developing systems that address the research task. A team consists of one or more members, from one or more collaborating institutes that can be academic or industrial.
The organizers define the research task clearly and provide measures to evaluate the performance of the systems. More importantly, the organizers provide datasets to the participants to help develop their systems. The participating teams can then use the datasets to train (and tune) their systems. At a later stage, a test set will be released on which the participating teams run their systems to get results and submit them to be evaluated by the organizers. The organizers then release the performance scores of the submitted results.
Can participating teams submit a research paper too?
Each participating team that submitted results (called runs) to our Qur'an QA shared task is allowed (and encouraged, but not required) to submit a research paper describing their system and analyzing its performance to OSACT5 workshop in a special track dedicated to our shared task. The submitted paper will be peer-reviewed, and, if accepted, it will be published within the workshop proceedings.
Can a participating team develop multiple systems?
Yes, of course. Our field is experimental, and you probably need to experiment with multiple ideas. So, one team can submit up to 3 runs for the test set. All of them will be evaluated by the organizers.
What is expected from the participating teams?
Subscribe to our discussion group to be updated on our shared task.
Download the training and dev set when they are released.
Develop a system that produces results in the given "run file format".
Once the test set is released, run the system on it, get results, submit them on our CodaLab site.
Write a research paper about the developed system and its performance. Submit the paper. This is strongly encouraged, but not required.
If the paper is accepted, you can register at LREC 2022 and go present your work at our OSACT5 workshop in Marseille, France, on the 20th of June 2022.