Overview
One script per subtask — task1_scoring.py, task2_scoring.py, task3_scoring.py, and task4_scoring.py — all run the same way.
Open the script and set ROOT_DIR at the top to a folder on your machine (or set the SCORING_ROOT environment variable to it). Inside that folder, create three subfolders:
input/ref/ the gold answer files
input/res/ your prediction file (one .tsv)
output/ the script writes the result here
What to put in each folder
input/res/ : your prediction .tsv for that subtask (columns as in the Submission Format section). Put exactly one .tsv file here.
input/ref/ : the gold answer TSV for that subtask (a single .tsv file, any name). For Subtask 1 only, also place the response file (the .jsonl with id and generated_answer) in this same folder, because the script needs it to measure each response's length. Subtasks 2 and 3 need only the gold TSV.
python task1_scoring.py
python task2_scoring.py
python task3_scoring.py
python task4_scoring.py
Note: Each script prints the score and writes it to output/scores.json.
Output
Subtask 1: {"F1 Score": number} — character-level macro-F1 over five classes (neither, Ayah, matn, isnad, claimed_source), pooled across the set.
Subtask 2: {"accuracy": macro, "accuracy_matn": ..., "accuracy_Ayah": ..., "accuracy_isnad": ..., "accuracy_claimed_source": ...} — accuracy per segment type, gold-N/A excluded, macro-averaged.
Subtask 3: {"accuracy": ..., "accuracy_Ayah": ..., "accuracy_matn": ...} — overall accuracy across all scored segments, plus per-type accuracy for Qur'an (Ayah) and Hadith (matn).
TODO
Columns: Response_ID, Annotation_ID, Segment_Type, Correction
You are given the incorrect segments. Provide the corrected text for each. Segment_Type is Ayah (Quran) or matn (Hadith). Correction is the authentic canonical text — the full verse for Quran, or the matn for Hadith — or the string خطأ when the span cannot be grounded to any authentic source. For example:
R000033 1 Ayah يَا أَيُّهَا الَّذِينَ آمَنُوا لَا تَدْخُلُوا بُيُوتًا غَيْرَ بُيُوتِكُمْ حَتَّىٰ تَسْتَأْنِسُوا وَتُسَلِّمُوا عَلَىٰ أَهْلِهَا ۚ ذَٰلِكُمْ خَيْرٌ لَكُمْ لَعَلَّكُمْ تَذَكَّرُونَ
R000033 2 matn إِذَا شَكَّ أَحَدُكُمْ فِي الصَّلَاةِ فَلْيَتَحَرَّ الصَّوَابَ ، ثُمَّ يَسْجُدْ سَجْدَتَيْنِ
R000033 3 matn خطأ
Provide exactly one correction per row. Although the gold may hold several acceptable corrections for a segment, you submit only one; it is counted correct if it matches any of the accepted corrections. In the above example, span 2 holds 4 accepted variants of this matn — choosing any of the 4, just as above, is considered correct.
Notes:
Only incorrect Ayah/matn segments are scored. Correct segments, isnad, claimed_source, and no-citation responses do NOT require rows in Subtask 3.
Rows are matched to the gold by (Response_ID, Annotation_ID). A missing prediction for a scored segment counts as incorrect.
Columns: question_id Response_ID Annotation_ID span_type span_text relevance_label
Given a user question and an LLM response with its correctly extracted Qur'anic and Hadith citation spans, the task is to determine whether each citation span is relevant to answering the question.
The question_id, Response_ID, and Annotation_ID fields link to the main JSONL file, which contains the full text of the user's question and the LLM's response.
span_type identifies the citation as either an Ayah or a matn.
span_text provides the correct text of the citation.
The task is to predict the relevance_label for each span using a binary classification: Relevant (1) or Non-relevant (0). Refer to the detailed relevance label definitions published at Subtask 4 webpage https://sites.google.com/view/islamiceval2026/subtask-4
Example
Q07863 R000017 1 Ayah وَمَا خَلَقْتُ الْجِنَّ وَالْإِنْسَ إِلَّا لِيَعْبُدُونِ 0
Q07863 R000017 2 matn إنما بعثت لأتمم مكارم الأخلاق 1