Datasets Description

The datasets for Subtask 2 are composed of the following collections and datasets.

• The Qur'anic Passage collection (QPC) [1,4].

• The Sahih Al-Bukhari collection [2].

• The questions of the AyaTEC dataset [3] and their relevance judgments over the Qur'anic Passage collection only.

The QPC was developed by topically segmenting the 114 Qur'anic chapters of different lengths using the Thematic Holy Qur'an [1], which is a printed edition that clusters the verses of each chapter into topics. This segmentation resulted in a total of 1,266 passages. The figure below exhibits two pages from the Thematic Qur'an visually segmented by color into four themes/topics according to their respective descriptions. 

The Sahih Al-Bukhari collection comprises 2,254 Hadiths, from which all redundant Hadiths and Arabic commentary have been excluded by the authors  [2].

The AyaTEC dataset comprises 250 questions, which are divided into training (84%) and development (16%) sets.  An additional set of 50 new questions is being developed for the test set, which will be used to evaluate the participating systems in Subtask 2. To make the retrieval task more realistic (thus challenging), we have included 37 questions (15%) that do not have an answer in the Holy Qur’an; these are referred to as zero-answer questions. The new test set will also include questions that do not have an answer in the Qur'an and/or Sahih Al-Bukhari Hadith collection. The query relevance judgments (QRels) are composed of 1,559 gold (answer-bearing) Qur'anic passage-ids considered relevant to each question.  For zero-answer questions, the passage-id will have a value of "-1".  Since the Hadith QA component of Subtask 2 is newly introduced in this year's shared task, participating teams are encouraged to utilize any existing Hadith QA resources for training their models and systems.