Subtask 4: Answer Relevance

Task Definition

Given a user question in Arabic, an LLM response to that question, and the correctly extracted Qur'anic and Hadith-matn citation spans from that response, classify each citation using a binary label: relevant (1) or non-relevant (0). A citation span is considered relevant if it directly or indirectly answers the question. Citation spans that are merely topically related but do not answer the question, as well as citation spans that are entirely unrelated, should be classified as non-relevant. A citation span may be authentic and accurately quoted yet still be non-relevant if it does not answer the question. 

Relevance Label Definitions:

• Relevant (1)

  • • • with direct answer: The citation span explicitly answers the question, and its meaning aligns with the question’s intent and context, or

      • with indirect answer: This applies in two cases:

        • Case 1: The citation span does not explicitly answer the question, but it implicitly provides the requested information, and its meaning aligns with the question’s intent and context.

        • Case 2: The citation span does not directly or implicitly answer the question on its own, but it provides clear textual support for the reasoning in the system’s response such that an answer to the question can be inferred.

• Non-relevant [Not an answer] (0): The citation span does not provide a direct or indirect answer to the question. If a citation is only topically related but does not actually answer the question or support the answer reasoning in a meaningful way, it must be marked as non-relevant. Topical overlap alone is insufficient.  

Partial support note: A relevant citation span may provide either:

o   a complete answer to the question, or

o   an important and necessary part of the answer.

Participating systems are limited to models of 13B parameters or fewer.

Evaluation Measures

The submissions will be evaluated using Macro-F1 per question, then averaged across all questions.