Overview
Submit one tab-separated (TSV) file per subtask, with a header row. Each row describes one segment of one citation. Rows are matched to the gold by the keys described below. The order of rows does not matter. However, a missing prediction for a scored segment counts as incorrect.
Columns: Response_ID, Annotation_ID, Segment_Type, Span_Start, Span_End
Segment_Type is one of: Ayah, matn, isnad, claimed_source. Spans are character offsets into generated_answer, end-exclusive. Each row indicates one segment. For example:
R000428 1 Ayah 120 168
R000428 2 matn 305 540
R000428 2 isnad 270 304
R000428 2 claimed_source 542 560
This says response R000428 contains two citations: citation 1 is a Qur'anic quotation (Ayah) spanning characters 120–168, and citation 2 is a Hadith whose isnad (270–304), matn (305–540), and claimed_source (542–560) were identified.
For a response that cites nothing, provide exactly one row with NoAnnotation and a dash for both spans:
R000322 1 NoAnnotation - -
This indicates the system found no Qur'an or Hadith citation in response R000322.
Columns: Response_ID, Annotation_ID, Segment_Type, Label
Label is one of: correct, incorrect. You are given the segment spans; predict a label for each. For example:
R000428 1 Ayah correct
R000428 2 matn incorrect
R000428 2 isnad correct
R000428 2 claimed_source correct
This says the Ayah of citation 1 is correct, and for citation 2 the matn is incorrect while its isnad and claimed_source are judged correct.
Notes:
Do not submit N/A. Segments that are not applicable are handled in the gold and excluded from scoring automatically.
Responses with no citations have no segments and require no rows in Subtask 2.
Columns: Response_ID, Annotation_ID, Segment_Type, Correction
You are given the incorrect segments. Provide the corrected text for each. Segment_Type is Ayah (Quran) or matn (Hadith). Correction is the authentic canonical text — the full verse for Quran, or the matn for Hadith — or the string خطأ when the span cannot be grounded to any authentic source. For example:
R000033 1 Ayah يَا أَيُّهَا الَّذِينَ آمَنُوا لَا تَدْخُلُوا بُيُوتًا غَيْرَ بُيُوتِكُمْ حَتَّىٰ تَسْتَأْنِسُوا وَتُسَلِّمُوا عَلَىٰ أَهْلِهَا ۚ ذَٰلِكُمْ خَيْرٌ لَكُمْ لَعَلَّكُمْ تَذَكَّرُونَ
R000033 2 matn إِذَا شَكَّ أَحَدُكُمْ فِي الصَّلَاةِ فَلْيَتَحَرَّ الصَّوَابَ ، ثُمَّ يَسْجُدْ سَجْدَتَيْنِ
R000033 3 matn خطأ
Provide exactly one correction per row. Although the gold may hold several acceptable corrections for a segment, you submit only one; it is counted correct if it matches any of the accepted corrections. In the above example, span 2 holds 4 accepted variants of this matn — choosing any of the 4, just as above, is considered correct.
Notes:
Only incorrect Ayah/matn segments are scored. Correct segments, isnad, claimed_source, and no-citation responses do NOT require rows in Subtask 3.
Rows are matched to the gold by (Response_ID, Annotation_ID). A missing prediction for a scored segment counts as incorrect.
Columns: question_id Response_ID Annotation_ID span_type span_text relevance_label
Given a user question and an LLM response with its correctly extracted Qur'anic and Hadith citation spans, the task is to determine whether each citation span is relevant to answering the question.
The question_id, Response_ID, and Annotation_ID fields link to the main JSONL file, which contains the full text of the user's question and the LLM's response.
span_type identifies the citation as either an Ayah or a matn.
span_text provides the correct text of the citation.
The task is to predict the relevance_label for each span using a binary classification: Relevant (1) or Non-relevant (0). Refer to the detailed relevance label definitions published at Subtask 4 webpage https://sites.google.com/view/islamiceval2026/subtask-4
Example
Q07863 R000017 1 Ayah وَمَا خَلَقْتُ الْجِنَّ وَالْإِنْسَ إِلَّا لِيَعْبُدُونِ 0
Q07863 R000017 2 matn إنما بعثت لأتمم مكارم الأخلاق 1