Embedded Files
Task Definition
Task Definition
Given an LLM response to a user question, for each incorrect Ayah or Hadith matn from Subtask 2, return the correct canonical text. Covers Ayahs and matns only. Complete verses and hadiths are expected, with writing and diacritics taken from the shared sources.
Participating systems are limited to models of 13B parameters or fewer.
Example
Example
In the example shown, the user is asking about the meaning of tawḥīd and its three types. We show the question along with the LLM response. The LLM response is citing Qur'anic verses and hadiths. There are two of its claimed citations — one Ayah (with its claimed source) and one Hadith (with its isnad and matn). In Subtask 2, we showed that the cited Ayah is incorrect. So the required correction should be as follows:
Evaluation Measures
Evaluation Measures
The correction will be measured using an exact match against the reference canonical text.
Page updated
Google Sites
Report abuse