Overview

Investigative journalists and volunteers have been working hard trying to get to the root of a claim and to present solid evidence in favor or against it. However, manual fact-checking is very time-consuming, and thus automatic methods have been proposed as a way to speed-up the process. For instance, there has been work on checking the factuality/credibility of a claim, of a news article, or of an entire news outlet. However, less attention has been paid to other steps of the fact-checking pipeline, e.g., check worthiness estimation has been severely understudied as a problem.

A typical fact-checking pipeline includes the following steps.

First check-worthy text fragments are identified.
Then, documents that might be useful for fact-checking the claim are retrieved from various sources and supporting evidence is extracted.
By comparing a claim against the retrieved evidence, a system can determine whether the claim is likely true or likely false (or unsure, if no supporting evidence either way could be found).

CheckThat! aims to address these understudied aspects. It fosters the development of technology capable of spotting check-worthy claims in English political debates in addition to providing evidence-supported verification of Arabic claims.

This lab is the second edition of the CheckThat! lab; the first was in CLEF 2018. Datasets and tools from last year's edition can be found here. Presentations of the participated teams summarizing their approaches and results can be found here.

Tasks

Task 1 - Check-Worthiness: Given a political debate or a transcribed speech, segmented into sentences, with speakers annotated, identify which sentence should be prioritized for fact-checking. This is a ranking task and systems are required to produce a score per sentence, according to which the ranking will be performed. This task will be run in English.

Task 2 - Evidence and factuality: Given a claim associated with a set of Web pages P (that constitute the results of Web search in response to using the claim as a search query), identify which of the Web pages (and passages of those Web pages) can be useful in assisting a human who is fact-checking the claim. Finally, judge the claim factuality according to the supporting information in the passages of the Web documents. This task will be run in Arabic.

Example Usage Scenario

Automated systems for claim identification and verification could be very useful as supportive technology for investigative journalism. They provide assistance and guidance and save time. A system could automatically identify check-worthy claims and present them to the journalist as a ranking from more to less relevant. Additionally, for a claim, the system could identify documents that are useful for humans to manually fact-check and it could also estimate a veracity score supported by evidence extracted from such documents, which would help the journalist to focus on the most outstanding cases. Another useful scenario, with the potential of impacting larger communities, would be helping the social media users who get a large flow of claims daily and want help in verifying them.