Run Submission

Run File

The participating team is required to write answers to all questions (of the dev or the test sets) in one file in a specific format. We call it a "run file" or a "run" in short. Each team is allowed to submit 30 runs on the dev set, but up to 3 runs on the test set. Each run typically constitutes the results of a different system or a model.

The name of each submitted run file should follow the below naming format.

<TeamID_RunID.json>

such that:

• • • • • TeamID can be an alphanumeric with a length between 3 and 9 characters

        • RunID can be an alphanumeric with a length between 2 and 9 characters "

For example, bigIR_run01.json.

Format of the Run File

The expected run file is in JSON format. It has a list of passage-question ids (pq_id) along with their respective ranked lists of returned answers. For each passage-question pair, the system should return up to 5 predicted answers, along with their ranks and estimated scores. Only the ranks are used in the evaluation (not the estimated scores). The run file format is shown below for a sample of two questions.

Download the Submission Checker Script

The run file submission checker script is released on our main repo.

Leaderboard and Submission Site

Having done the registration steps mentioned here, you can submit your run by sticking to the following steps.

How to submit your runs

The following steps should be done by the team leader only as we will approve the team leader only in Codalab

1. Make sure that the name of your run file adopts this naming format described above. You can use the submission checker script to achieve that.

2. Convert your run file into a .zip file.

3. Login to CodaLab.

4. Go to our "Qur'an QA 2022 Shared Task".

5. Go to "Participate" tab, then "Submit/ View Results" tab. Then, you can upload your run file by clicking on "Submit". Optionally, you can add a description of your submission before uploading the file. If your run fails, check the error message below the "Submit" button.

6. Leaderboard results of your run (in case no errors in the run file) will be shown in the "Results" tab. Only your team's best run will be shown. However, you can still see all the history of your submitted runs in "Submit/ View Results" tab.

Number of runs: Please note that in the development phase, you can submit up to 30 runs. However, in the testing phase, you are allowed to submit 3 runs only. In both phases, the best run will be shown on the leaderboard.

Baseline: The run shown in the leaderboard under username watheq9 is a simple baseline that just answers each question by giving the full passage. It is there just as a reference point.