Subtask 2: Qur'an and Hadith QA 2025

Task Definition

This subtask is a continuation of Qur'an QA 2022 and Qur'an QA 2023 shared tasks.

The subtask this year is defined as follows: Given a free-text question posed in MSA, a collection of Qur'anic passages (that cover the Holy Qur'an) and a collection of Hadiths from Sahih Bukhari, a system is required to retrieve a ranked list of up-to 20 answer-bearing Qur'anic passages or Hadiths (i.e., Islamic sources that potentially enclose the answer(s) to the given question) from the two collections. The question can be a factoid or non-factoid question.

To make the task more realistic (thus challenging), some questions may not have an answer in the Holy Qur'an and Sahih Al-Bukhari. In such cases, the ideal system should return no answers; otherwise, it returns a ranked list of up to 20 answer-bearing sources.

Here are sample questions and answers.

Evaluation Measures

As the task is a classical ranked retrieval task, Mean Average Precision (MAP) will be used as the main official measure for evaluation. The no answer cases will be handled simply by giving full credit to "no answers'' system output and zero otherwise. We will report 3 measures:

MAP@10: computed over the top 10 ranked answers.
MAP_Q@5: computed over the top 5 ranked Qur'anic passages (i.e., after discarding all ranked Hadiths).
MAP_H@5: computed over the top 5 ranked Hadiths (i.e., after discarding all ranked Qur'anic passages).

Registration

Dataset

Detailed information about the dataset is here.

Run Submission

Detailed information for formatting and submitting your runs is here.

Please submit your runs in the following link

Page updated

Google Sites

Report abuse