Shared Task Details

Shared Task Objectives

The shared task aims to serve as a collaborative platform where participants propose guidelines and diverse methods for annotating and analyzing the dataset. There will be two subtasks of focus:

• Subtask on Bias Annotation

7 possible Labels:

1. Unbiased

2. Biased against Palestine

3. Biased against Israel

4. Biased against both Palestine and Israel

5. Biased against others

6. Unclear

7. Not Applicable

Bias Annotation Examples to illustrate the 7 Labels.

1. Unbiased:

Example: "In the ongoing Israel-Palestine conflict, recent events have escalated tensions. Yesterday, Israeli forces conducted operations in response to rocket attacks from Gaza. Both sides have reported casualties. International leaders are calling for restraint and a return to peace talks."

2. Biased against Palestine:

Example: "Once again, Palestinian aggression has disrupted peace in the region. Palestinian extremists, ignoring efforts for peace, launched unprovoked attacks on innocent Israeli civilians. Israel's response, though portrayed as harsh by some, is a justified measure to protect its citizens."

3. Biased against Israel:

Example: "In a typical display of excessive force, Israeli troops have yet again targeted Palestinian areas, causing numerous civilian casualties. This aggression, under the guise of self-defense, highlights the ongoing oppressive tactics Israel employs against Palestinians."

4. Biased against both Palestine and Israel:

Example: "In the latest chapter of their endless and futile conflict, Israeli and Palestinian forces have once again engaged in senseless violence. Both sides continue to commit atrocities, showing a complete disregard for peace or human life."

5. Biased against others:

Example: "In the shadow of the Israel-Palestine conflict, external actors, particularly Iran, are exacerbating tensions. Iran's covert support for extremist groups shows its intent to destabilize the region, disregarding the catastrophic impact on both Israeli and Palestinian civilians."

6. Unclear:

Example: "Recent developments in the Middle East have seen an increase in hostilities. The situation in the region is complex, with various factors contributing to the current state of affairs. The international community remains divided on the issue."

7. Not Applicable:

Example: "In other news, the annual technology conference in Tel Aviv has unveiled groundbreaking advancements in cybersecurity. Industry leaders from around the globe gathered to showcase innovations that promise to shape the future of digital security."

• Subtask on Propaganda Annotation

4 possible Labels:

• Propaganda

• Not Propaganda

• Unclear

• Not Applicable

Propaganda Annotation Examples to illustrate the 4 possible categories.

1. Propaganda

Example: "In a display of unmatched heroism, our troops have once again safeguarded our nation from the brink of destruction, heroically neutralizing the threat from Gaza, which aims to undermine our very existence."

2. Not Propaganda:

Example: "Yesterday, an escalation occurred along the Israel-Gaza border, resulting in casualties on both sides. Israeli and Palestinian officials provided conflicting accounts of the events that led to the confrontation."

3. Unclear:

Example: "The situation in Gaza remains tense, with reports of civilian distress and military movements. While some sources claim the military actions are defensive, others argue they are provocative, leaving the true nature of the situation open to interpretation."

4. Not Applicable:

Example: "A feature on Gaza's cultural scene highlights the resilience of its art community, showcasing how local artists use their craft to express hope and endurance amid challenging circumstances, without delving into the political context."

For each subtask, there will be three evaluation tracks for which winners will be crowned.

Guidelines Track:  The teams have the freedom to design their own guidelines and apply them to the shared data. The following is the checklist of all items that will be evaluated by the organizers.

• Detailed annotation guidelines including examples for all main and corner cases.
  Consider the following subtasks:

  • Define the Objective: Outline the purpose and specific NLP task.

  • Describe the Task: Provide a detailed task description with correct examples.

  • Establish Categories: List and define all annotation labels/categories/tags.

  • Detailed Category Guidelines: Explain application criteria for each category/tag, with examples.

  • Include Examples: Offer examples for correct application and common mistakes.

  • Outline the Process: Describe the step-by-step annotation process and tools used.

  • Set Quality Standards: Define expectations for accuracy and consistency, along with quality check procedures.

  • Handle Ambiguities: Provide guidance on ambiguous cases and a protocol for seeking clarification.

  • Ensure Consistency: Implement measures for annotator consistency and recommend calibration sessions.

  • Ethical Considerations: Highlight unbiased annotation practices and handling of sensitive data.

  • Training and Support: Detail training procedures and support resources for annotators.

  • Review and Update: Schedule guideline reviews for updates based on feedback and new insights.

  • Feedback Mechanism: Include a system for annotator feedback to refine guidelines and processes.

• Annotate a minimum of two batches (1,800 posts) is needed and their designated Inter-annotator agreement subset (200 posts) for a total of 2000 posts (from two batches). The IAA subset must be done by every annotator on the team; but the rest can be divided among them.

• Each 1000-post batch will contain 200 posts from 5 languages: Arabic, English, French, Hebrew, and Hindi, together with their machine translated versions into Arabic and English (as needed).  Participants must specify whether they annotated the original language or its machine translated version. So a monolingual Arabic team can annotate the full batch in Arabic (original or translated).

• The batches will be provided to the annotation teams with clear instruction on how to submit the results.

 The teams must provide well-documented annotation guidelines including examples, and must provide inter-annotator agreement (IAA) numbers for at least 200 posts (40 from each language) from Batch 1 and Batch 2.  We expect the IAA to be competitive (e.g. Cohen Kappa of 0.6+) in the target space. The best guidelines will be selected by the organizers.  Furthermore, Guidelines and IAA are necessary conditions to participate in the Quantity and Consistency Tracks (see below).  

Quantity Track: The teams can compete in the number of annotated data batches. They must finish them in order and complete a batch before moving to the next.  The teams with the highest number of completed batches will be crowned the Quantity track winners for the subtask of choice. Please note that correlation measure will also be measured for the quantity track and will be used during the evaluation.

Consistency Track: The various teams in the same subtask and shared completed batches will be compared for correlation against each other. The teams that have the highest correlation against other teams (centroidal choices) will be crowned winner. This needs a minimum of three teams per subtask. 

Publication: All teams participating in the Shared Task are invited to submit short paper (4 pages) descriptions of their efforts. These papers will be evaluated by multiple reviewers to be selected for publication in the Arabic NLP 2024 Conference Proceedings and indexed by the ACL Anthology.

Collaborative Commitment: Participants are encouraged to join the shared task with a commitment to collaboration. Whether working independently or within teams, every effort and insight contributed should be shared openly. This collaborative ethos extends beyond individual tasks and includes sharing methodologies, findings, and results.

Optional Demographic Details: We would like to invite participants to provide some demographic details voluntarily. This information includes aspects such as age range, native language, educational background, area of study or expertise, gender, and region of origin. Please note that providing this demographic information is entirely optional and will not influence the evaluation of your participation in any way. We respect your privacy and understand if you choose not to share these details.