The participating team is required to submit one file in a specific format. We call it a "run file" or a "run" in short. Each team is allowed to submit 30 runs on the dev set, but up to 3 runs on the test set. Each run typically constitutes the results of a different system or a model.
The name of each submitted run file should follow the below naming format.
<TeamID_RunID.tsv>
such that:
TeamID can be an alphanumeric with a length between 3 and 9 characters
RunID can be an alphanumeric with a length between 2 and 9 characters "
For example, kaust_bm25.txt.
The run file should match the TREC run format, i.e., having the following columns: ["question-id", "Q0", "passage-id" or "hadith-id", "rank", "score", "tag"]. Each row indicates one retrieved passage or hadith. The expected run file is in tsv format (tab separated). The run file has a list of question-ids along with their respective ranked lists of retrieved passages.
For example:
428 Q0 q-125 3 5.63 kaust_bm25indicates that the system retrieved the Qur'anic passage q-125 at rank 3 with a score 5.63 for the question 428. The "Q0" value is just there for backward compatibility (please just add it as is).
For the zero-answer questions, you need to provide one (and only one) row for each zero-answer question with "-1" in the "passage-id" column and "1" in the rank column with any score. For example
123 Q0 -1 1 0.45 kaust_bm25indicates that the system found no answer for question 123. Here, we added -1 in the "passage-id" column and 1 in the rank column.
The run file format is shown below for a sample of three questions (two of which are zero-answer questions).