Task A Run Submission

Run File

The participating team is required to submit one file in a specific format. We call it a "run file" or a "run" in short. Each team is allowed to submit 30 runs on the dev set, but up to 3 runs on the test set. Each run typically constitutes the results of a different system or a model.

The name of each submitted run file should follow the below naming format.

<TeamID_RunID.tsv>

such that:

- - - - TeamID can be an alphanumeric with a length between 3 and 9 characters
      - RunID can be an alphanumeric with a length between 2 and 9 characters "

For example, bigIR_bm25.txt.

Format of the Run File

The run file should match the TREC run format, i.e., having the following columns: ["question-id", "Q0", "passage-id", "rank", "score", "tag"]. Each row indicates one retrieved passage. The expected run file is in tsv format (tab separated). The run file has a list of question-ids along with their respective ranked lists of retrieved passages.

For example:

428 Q0 6:22-26 3 5.63 bigIR_bm25

indicates that the system retrieved the passage 6:22-26 at rank 3 with a score 5.63 for the question 428. The "Q0" value is just there for backward compatibility (please just add it as is).

For the zero-answer questions, you need to provide one (and only one) row for each zero-answer question with "-1" in the "passage-id" column and "1" in the rank column with any score. For example

123 Q0 -1 1 0.45 bigIR_bm25

indicates that the system found no answer for question 123. Here, we added -1 in the "passage-id" column and 1 in the rank column.

The run file format is shown below for a sample of three questions (two of which are zero-answer questions).

428 Q0 6:22-26 1 10.5 ntir_run01428 Q0 7:179-179 2 9.67 ntir_run01428 Q0 22:42-46 3 5.4 ntir_run01428 Q0 47:20-24 4 1.04 ntir_run01322 Q0 -1 1 2.3 ntir_run01260 Q0 -1 1 6.9 ntir_run01

Download the Submission Checker Script

The run file submission checker script is released on our main repo. It is mandatory to use this script to verify your run file (prior to submission).

Leaderboard and Submission Site

Having done the registration steps mentioned here, you can submit your run by sticking to the following steps.

How to submit your runs

The following steps should be done by the team leader only as we will approve the team leader only in Codalab

Make sure that the name of your run file adopts this naming format described above. You can use the submission checker script to achieve that.
Convert your run file into a .zip file.
Login to CodaLab.
Go to our "Qur'an QA 2023 - Task A (Passage Retrieval) ".
Go to "Participate" tab, then "Submit/ View Results" tab. Then, you can upload your run file by clicking on "Submit". Optionally, you can add a description of your submission before uploading the file. If your run fails, check the error message below the "Submit" button.
Leaderboard results of your run (in case no errors in the run file) will be shown in the "Results" tab. Only your team's best run will be shown. However, you can still see all the history of your submitted runs in "Submit/ View Results" tab.

Number of runs: Please note that in the development phase, you can submit up to 30 runs. However, in the testing phase, you are allowed to submit 3 runs only. In both phases, the best run will be shown on the leaderboard.

Baseline: The run shown in the leaderboard under username watheq9 is the BM25 baseline.