Task Definition: Given a check-worthy claim, and a set of previously-checked claims, determine whether the claim has been previously fact-checked with re- spect to a collection of fact-checked claims.
Subtask 2A: Detect previously fact-checked claims from tweets: Given a tweet, detect whether the claim the tweet makes was previously fact-checked with respect to a collection of previously fact-checked claims. This is a ranking task, where the systems will be asked to produce a list of top-n candidates. This task is given in two languages:
Arabic
English
Subtask 2B: Detect previously fact-checked claims in political debates/speeches: Within the context of a political debate or a speech, detect whether a claim has been previously fact-checked with respect to a collection of previously fact-checked claims. This is a ranking task, where systems are asked to produce a list of top-n candidates. This task is only given in English.
This task is evaluated as a ranking task. Ranked list per claim will be evaluated using ranking evaluation measures (MAP@k for k=1,3,5,10,20,all, MRR and Precision@k for k=1,3,5,10,20). Official measure is MAP@5.
Subtask-2a--Arabic: https://gitlab.com/checkthat_lab/clef2022-checkthat-lab/clef2022-checkthat-lab/-/tree/main/task2/data/subtask-2a--arabic
Subtask-2a--English: https://gitlab.com/checkthat_lab/clef2022-checkthat-lab/clef2022-checkthat-lab/-/tree/main/task2/data/subtask-2a--english
Subtask-2b--English: https://gitlab.com/checkthat_lab/clef2022-checkthat-lab/clef2022-checkthat-lab/-/tree/main/task2/data/subtask-2b--english
All scripts can be found on the main git repo for the lab, CheckThat! Lab Task 2: https://gitlab.com/checkthat_lab/clef2022-checkthat-lab/clef2022-checkthat-lab/-/tree/main/task2
Make sure that you create one account for each team, and submit it through one account only.
We will keep the leaderboard private till the end of the submission period, hence, results will not be available upon submission. All results will be available after the evaluation period.
You are allowed to submit max 200 submissions per day for each subtask.
The last file submitted to the leaderboard will be considered as the final submission.
You are allowed to submit max 200 submissions per day for each subtask.
The last file submitted to the leaderboard will be considered as the final submission.
The output file has to have a `.tsv` extension; otherwise, you will get an error on the leaderboard.
You have to zip the tsv, `zip submission.zip path_to_tsv_file.tsv` and submit it through the codalab page.
Please submit your results on test data here: https://codalab.lisn.upsaclay.fr/competitions/4260