Participants must submit their system predictions in JSON format. The submitted JSON file must strictly follow the specified schema. Any deviation from the required format may result in the submission being rejected by the evaluation script.
The submission file must be a valid JSON file.
The submission file must be a valid JSON file.
The JSON file must contain exactly one prediction for every claim in the test set.
Every test instance must appear exactly once.
The ID field must exactly match the corresponding claim ID provided in the test set.
Do not modify, remove, or reorder the claim IDs.
All files must be encoded using UTF-8.
For each claim, participants must retrieve the top-3 ranked evidences from the provided evidence corpus and the veracity label.
Submission Format:
The submission file must be a JSON array, where each element contains:
ID
top-3-evidences (evidence text)
Veracity label
Example:
[
{
"ID": "S1/4530",
"top_3_evidence": [
"Evidence 1",
"Evidence 2",
"Evidence 3"
],
"Prediction": "SUPPORTS"
},
{
"ID": "S1/4968",
"top_3_evidence": [
"Evidence 1",
"Evidence 2",
"Evidence 3"
],
"Prediction": "REFUTES"
}
]
Requirements:
Each ID must appear exactly once.
top_3_evidence must contain exactly three evidence texts.
The evidences must be ranked from most relevant to least relevant.
The evidence text should exactly match the retrieved evidence from the provided evidence corpus.
Empty evidence lists are not permitted.
Every prediction must be either SUPPORTS or REFUTES.
Veracity labels are case-sensitive and must be written exactly as specified.
In this subtask, participants are provided only the claim. Systems are expected to retrieve relevant evidence from open-source knowledge repositories (e.g., Wikipedia, publicly available web pages, news articles, or other reliable open-domain resources) and determine the veracity of the claim.
The submission file must be a valid UTF-8 encoded JSON file and must strictly follow the format described below.
Submission Format:
The submission file must be a JSON array. Each element of the array corresponds to one claim and must contain the following fields:
ID: Unique claim identifier.
Evidence: A structured and concise evidence summary retrieved from external sources.
Prediction: Predicted veracity label. (SUPPORTS/ REFUTES)
Justification: A concise explanation supporting the predicted label (within 100-120 words).
Only the following labels are permitted: SUPPORTS and REFUTES. Labels are case-sensitive and must exactly match the specified values.
Example:
[
{
"ID": "S2/4530",
"Evidence": "<structured and concise evidence summary retrieved from external sources >"
"Prediction": "SUPPORTS",
"Justification": <concise explanation within 100-120 words supporting the predicted label>,
},
{
"ID": "S2/4968",
"Evidence": "<structured and concise evidence summary retrieved from external sources >",
"Prediction": "REFUTES",
"Justification": <concise explanation within 100-120 words supporting the predicted label>
}
]
Important Notes:
The submitted evidence must be grounded in retrieved information from external sources.
Purely LLM-generated evidence without retrieval support is not permitted.
Systems may use Large Language Models to summarize or organize retrieved evidence, provided that the generated content faithfully reflects the retrieved sources.
Participants are encouraged to preserve factual accuracy and avoid hallucinated information.
Submission Validation:
Invalid JSON format.
File is not UTF-8 encoded.
Missing claim IDs.
Unknown claim IDs.
Duplicate claim IDs.
Missing required fields (Evidence, Prediction, or Justification).
Invalid prediction labels.
Empty evidence or justification fields.
The official test set for each subtask is released separately for four languages: English, Hindi, Bengali, and CodeMix. Participants are required to submit a separate prediction file for each language. Therefore, each subtask submission must contain four JSON prediction files, one corresponding to each language-specific test set.
The recommended file naming convention is as follows: <language>_subtask<task_number>.json.
For example:
English_subtask1.json
Hindi_subtask1.json
Bengali_subtask1.json
CodeMix_subtask1.json
English_subtask2.json
Hindi_subtask2.json
Bengali_subtask2.json
CodeMix_subtask2.json
------------------------------------------------------------------------------------------------------------------------------------------------------
All participants must submit their prediction files and source code as a single compressed ZIP archive through the official Google Form. The submission link will be shared with all registered participants via email. A team can submit a maximum of three submissions.
The ZIP archive must contain the following:
Prediction files for Subtask 1 (Evidence Selection + Veracity Prediction, in JSON format) for each language.
Prediction file for Subtask 2 (Open-Domain Claim Verification, in JSON format) for each language.
Complete Python source code used for Subtask 1 (including both evidence selection and veracity prediction).
Complete Python source code used for Subtask 2.
Participants are encouraged to organize the contents of the ZIP archive in a clear and structured manner. The source code should be executable and include all necessary scripts required to reproduce the submitted predictions.
The ZIP archive should follow the following naming convention: TeamName.zip
For example: NLPResearchLab.zip
Submissions that are incomplete, incorrectly formatted, or submitted through channels other than the official Google Form may not be considered for evaluation.