Arabic LLMs Hallucination-Shared Task

Discussion Group

• Please join our Google discussion group (https://groups.google.com/g/arabicllmshallucination) to receive announcements and participate in discussions.

 Important Dates

All times are Anywhere On Earth (AOE).

• 01 March 2024: Shared-task paper submission deadline

• 25 March 2024: Notification of acceptance

• 30 March 2024: Camera-ready submission of manuscripts

• 25 May 2024: Workshop in Torino, Italy.

Submission of System Results:

Evaluation Criteria:

Classification systems will be evaluated using the macro-averaged F1-score for all subtasks.

Submission Format:

Classifications of test and dev datasets (claim_id and labels only) should be submitted as separate files in the following format with a label for each corresponding claim:

For Subtasks A and B:

    claim_id \t label (values are: FC, FI, or NF)\n

The first line in the submission file should contain the title in the next format, and it will be skipped during evaluation: 

"claim_id" \t "label" \n

Then, the following lines should contain the actual claim ids and the labels, ex:

4584 \t FC \n

283 \t FI \n

1474 \t NF \n

and so on.

Participants can submit up to two system results (primary submission for their best result, and a secondary submission for the 2nd best result).

Official results will consider primary submissions for ranking different teams, and results of secondary submissions will be reported for guidance. All participants are required to report on the development and test sets in their papers.

Submission filename should be in the following format:

ParticipantName_Subtask<A/B>_<test/dev>_<1/2>.zip (a plain .txt file inside each .zip file)

Ex: QCRI_SubtaskA_test_1.zip (the best results for Subtask A for test dataset from QCRI team)

Ex: KSU_SubtaskB_dev_2.zip (the 2nd best results for Subtask B for dev dataset from KSU team)

Paper Submission

Please submit your paper using START system

Content Guidelines

• Along with the paper, it is mandatory to submit the code that generated the submitted runs. Instructions on how to submit the code will be communicated later.

• The paper title should follow the following template: <TeamID> at Arabic LLMs Hallucination 2024: <Title>. An example title is "QCRI at Arabic LLMs Hallucinations 2024: Detect Hallucinations using Transformers".

• The paper should cover (among other standard sections) the following:

  • Approach: system overview, models, training, external resources, etc. Please make sure that you give enough details about your approach.

  • Experimental Evaluation: results on the dev set, official results on the test set, analysis/discussion of the results, etc. You can also report and analyze the results of other runs that you didn’t officially submit.

• For ease of approach reproducibility (and faster learning by others), you are strongly encouraged to release your code and make it publicly available. If so, please indicate that in your paper and provide a public link.

Formatting Guidelines

• Papers must use LREC template and comply to the LREC stylesheet.

• Papers must be written in English.

• Papers must be 4 to 8 pages long, excluding references and appendices (the number of appendix pages should remain reasonable though).

• All papers will be peer reviewed by at least two independent reviewers. The review is single-blind. So, the submissions are NOT anonymous, i.e., you need to add the author names on the first page of the submitted paper.

• Papers must be submitted electronically in PDF format.

• When submitting a paper at the START page, authors will be asked to provide essential information about resources (in a broad sense, i.e. also technologies, standards, evaluation kits, etc.) that have been used for the work described in the paper.

• All authors are encouraged to share the described LRs (i.e., language resources, such as data, tools, services, etc.) to enable their reuse and replicability of experiments (including evaluation ones).