Shared task on Propaganda Detection in Arabic

Welcome to Propaganda Detection in Arabic shared task at WANLP!

Propaganda is defined as an expression of opinion or action by individuals or groups deliberately designed to influence opinions or actions of other individuals or groups with reference to predetermined ends and this is achieved by means of well-defined rhetorical and psychological devices. Currently, propaganda (or persuasion) techniques have been commonly used on social media to manipulate or mislead social media users. More detail of the techniques can be found here: https://propaganda.qcri.org/annotations/definitions.html.

The goal of the shared task is to build models for identifying such techniques in the Arabic social media text (specifically Tweets).

Leaderboards

• Please check the leaderboards here: https://docs.google.com/spreadsheets/d/1Iu544EkD-i2QRtsiNO-PcvH1voI0mh55MtMvgqeTOIM/edit?usp=sharing

Lab Registration

• Please register here: https://shorturl.at/ftuz9

Important Dates

• July 24, 2022: Release of training, dev and dev-test data, and evaluation scripts.

• September 10, 2022: Registration deadline.

• September 10, 2022: Release of test data (and final training and dev data).

• September 15, 2022: End of the evaluation cycle (test set submission closes)

• September 17, 2022: End of the evaluation cycle (test set submission closes)

• September 20, 2022: Results released

• September 23, 2022: Results released

• October 5, 2022: System description paper submissions due.

• October 15, October 17, 2022: Notification of acceptance.

• October 31, 2022: Camera-ready versions due.

• December 7, 2022: WANLP 2022 workshop at EMNLP in Abu Dhabi

Submission

Submission link: https://codalab.lisn.upsaclay.fr/competitions/7274, please register to the codalab system to participate.

Submission instructions

• Make sure that you create one account for each team, and submit it through one account only.

• We will keep the leaderboard private till the end of the submission period, hence, results will not be available upon submission. All results will be available after the evaluation period.

• You are allowed to submit max 200 submissions per day for each subtask.

• The last file submitted to the leaderboard will be considered as the final submission.

• Name of the output file should with json extension (e.g., team_name_run_no_subtask1.json); otherwise, you will get an error on the leaderboard.

• You have to zip the json, `zip team_name_run_no_subtask1.zip team_name_run_no_subtask1.json` and submit it through the codalab submission page: https://codalab.lisn.upsaclay.fr/competitions/7274#participate

Instructions to prepare your paper

The title of paper should be in the following format:
<Team Name> at WANLP 2022 Shared Task: <Name of the title>

For example:
TeamX at WANLP 2022 Shared Task: Data Augmentation and Ensemble Models for Propaganda Detection in Arabic

More information about template and submission system: https://sites.google.com/view/propaganda-detection-in-arabic/paper-submission-guidelines

Tasks

We defined the following subtasks:

Subtask 1: Given the text of a tweet, identify the propaganda techniques used in it (multilabel classification problem).

Subtask 2: Given the text of a tweet, identify the propaganda techniques used in it together with the span(s) of text in which each propaganda technique appears. This is a sequence tagging task.

Evaluation Metrics

Subtask 1: The official evaluation metric for the task is micro-F1. However, the scorer also reports macro-F1.

Subtask 2: Task 2 is a multi-label sequence tagging task. We modify the standard micro-averaged F1 to account for partial matching between the spans.

Datasets

• The Datasets are hosted on the Gitlab repository: https://gitlab.com/arabic-nlp/propaganda-detection/

Organizers

• Firoj Alam, Qatar Computing Research Institute, HBKU

• Hamdy Mubarak, Qatar Computing Research Institute, HBKU

• Wajdi Zaghouani, HBKU

• Preslav Nakov, Mohamed bin Zayed University of Artificial Intelligence

• Giovanni Da San Martino, University of Padova

Citation

Please cite the following paper. In addition, in the following link (https://gitlab.com/arabic-nlp/propaganda-detection/) you can find a list of relevant work that might be useful to prepare your paper. Please see the section recommended reading.

We also shared bibliography files that might be useful: https://gitlab.com/arabic-nlp/propaganda-detection/-/tree/main/bibtex

@inproceedings{propaganda-detection:WANLP2022-overview,
title = "Overview of the {WANLP} 2022 Shared Task on Propaganda Detection in {A}rabic",
author = "Alam, Firoj and Mubarak, Hamdy and Zaghouani, Wajdi and Nakov, Preslav and Da San Martino, Giovanni",
booktitle = "Proceedings of the Seventh Arabic Natural Language Processing Workshop",
month = Dec,
year = "2022",
address = "Abu Dhabi, UAE",
publisher = "Association for Computational Linguistics",

}