Task 2: Detecting Previously Fact-Checked Claims

Definition

Task Definition: Given a check-worthy claim, and a set of previously-checked claims, determine whether the claim has been previously fact-checked with re- spect to a collection of fact-checked claims.

Subtask 2A: Detect previously fact-checked claims from tweets: Given a tweet, detect whether the claim the tweet makes was previously fact-checked with respect to a collection of previously fact-checked claims. This is a ranking task, where the systems will be asked to produce a list of top-n candidates. This task is given in two languages:

• Arabic

• English

Subtask 2B: Detect previously fact-checked claims in political debates/speeches: Within the context of a political debate or a speech, detect whether a claim has been previously fact-checked with respect to a collection of previously fact-checked claims. This is a ranking task, where systems are asked to produce a list of top-n candidates. This task is only given in English.

Evaluation

This task is evaluated as a ranking task. Ranked list per claim will be evaluated using ranking evaluation measures (MAP@k for k=1,3,5,10,20,all, MRR and Precision@k for k=1,3,5,10,20). Official measure is MAP@5.

Datasets

Subtask-2a--Arabic: https://gitlab.com/checkthat_lab/clef2022-checkthat-lab/clef2022-checkthat-lab/-/tree/main/task2/data/subtask-2a--arabic
Subtask-2a--English: https://gitlab.com/checkthat_lab/clef2022-checkthat-lab/clef2022-checkthat-lab/-/tree/main/task2/data/subtask-2a--english
Subtask-2b--English: https://gitlab.com/checkthat_lab/clef2022-checkthat-lab/clef2022-checkthat-lab/-/tree/main/task2/data/subtask-2b--english

Scorers, Format Checkers, and Baseline Scripts

All scripts can be found on the main git repo for the lab, CheckThat! Lab Task 2: https://gitlab.com/checkthat_lab/clef2022-checkthat-lab/clef2022-checkthat-lab/-/tree/main/task2

Submission Guidelines

• Make sure that you create one account for each team, and submit it through one account only.

• We will keep the leaderboard private till the end of the submission period, hence, results will not be available upon submission. All results will be available after the evaluation period.

• You are allowed to submit max 200 submissions per day for each subtask.

• The last file submitted to the leaderboard will be considered as the final submission.

• You are allowed to submit max 200 submissions per day for each subtask.

• The last file submitted to the leaderboard will be considered as the final submission.

• The output file has to have a `.tsv` extension; otherwise, you will get an error on the leaderboard.

• You have to zip the tsv, `zip submission.zip path_to_tsv_file.tsv` and submit it through the codalab page.

Leaderboard and Submission Site

Please submit your results on test data here: https://codalab.lisn.upsaclay.fr/competitions/4260