OffensEval 2020

This is the website of the second edition of OffensEval 2020: Multilingual Offensive Language Identification in Social Media organized at SemEval 2020 (Task 12).

The competition is now finished. For more information, please consult the OffensEval 2020 report. When referring to OffensEval 2020 please use the bib entry below. 

@inproceedings{zampieri-etal-2020-semeval,
    title = {{SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)}},
    author = {Zampieri, Marcos and Nakov, Preslav and Rosenthal, Sara and Atanasova, Pepa and Karadzhov, Georgi and Mubarak, Hamdy and Derczynski, Leon and Pitenis, Zeses and \c{C}\"{o}ltekin, \c{C}a\u{g}r{\i}},
    booktitle = {Proceedings of SemEval},
    year = {2020}
}


Description 

Offensive language is pervasive in social media. Individuals frequently take advantage of the perceived anonymity of computer-mediated communication, using this to engage in behavior that many of them would not consider in real life. Online communities, social media platforms, and technology companies have been investing heavily in ways to cope with offensive language to prevent abusive behavior in social media. One of the most effective strategies for tackling this problem is to use computational methods to identify offense, aggression, and hate speech in user-generated content (e.g. posts, comments, microblogs, etc.). 

This topic has attracted significant attention in recent years as evidenced in recent publications (Waseem et al. 2017; Davidson et al., 2017, Malmasi and Zampieri, 2018, Kumar et al. 2018) and workshops such as AWL and TRAC and competitions such as HatEval 2019 (Basile et al. 2019), HASOC 2019, and OffensEval 2019 (Zampieri et al. 2019).

The first OffensEval was organized at SemEval 2019OffensEval 2019 used the Offensive Language Identification dataset (OLID) a dataset containing English tweets annotated using a hierarchical three-level annotation model described in this paperNearly 800 teams signed up to participate in OffensEval 2019. The competition received more than 100 submissions across three sub-tasks.The findings are described in the OffensEval 2019 reportThe response received in 2019 by far exceeded our expectations and motivated us to organize OffensEval 2020. 


Data

OffensEval 2020 features a multilingual dataset with five languages. The languages included in OffensEval 2020 are:
  • Arabic
  • Danish
  • English
  • Greek
  • Turkish
The annotation follows the hierarchical tagset proposed in the Offensive Language Identification Dataset (OLID) and used in OffensEval 2019. In this taxonomy we break down offensive content into the following three sub-tasks taking the type and target of offensive content into account. The following sub-tasks will be organized:
  • Sub-task A - Offensive language identification;
  • Sub-task B - Automatic categorization of offense types;
  • Sub-task C - Offense target identification.


Important Dates

  • Trial data ready: July 31, 2019 (available here)
  • Training and development data: January 6, 2020
  • Test data available/Evaluation starts: Feb 20, 2020
    • Sub-task A: February 20 - March 4 (Arabic, Danish, Greek, and Turkish) (extended deadline)
    • Sub-task A: February 28 - March 11 (English only) (extended deadline)
    • Sub-task B: February 28 - March 4 (English only)
    • Sub-task C: March 6 - March 11 (English only)
  • Evaluation ends: March 11, 2020
  • Results announced: March 17, 2020 (available here)
  • Paper submission deadline: May 15, 2020
  • Notification to authors: June 24, 2020
  • Camera-ready due: July 8, 2020
  • SemEval workshop: December 12 and 13, 2020


Results

The test phase of OffensEval 2020 has come to an end. Thank you very for participating! 

The rankings are in the spreadsheets attached to this page. We are listing the usernames as they appear on CodaLab and the F1 score of your LAST submission. Please see the OffensEval 2020 report for more information.


Organizers

Marcos Zampieri - Rochester Institute of Technology, USA
Preslav Nakov - Qatar Computing Research Institute, Qatar
Sara Rosenthal - IBM Research, USA
Pepa Atanasova - University of Copenhagen, Denmark
Georgi Karadzhov - University of Cambridge, UK
Hamdy Mubarak - Qatar Computing Research Institute, Qatar
Leon Derczynski - IT University Copenhagen, Denmark
Zeses Pitenis - University of Wolverhampton, UK
Çağrı Çöltekin - University of Tübingen, Germany


References

Basile, V., Bosco, C., Fersini, E., Nozza, D., Patti, V., Pardo, F.M.R., Rosso, P. and Sanguinetti, M., (2019) Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation (pp. 54-63).

Davidson, T., Warmsley, D., Macy, M. and Weber, I. (2017) Automated Hate Speech Detection and the Problem of Offensive Language. Proceedings of ICWSM.

Kumar, R., Ojha, A.K., Malmasi, S. and Zampieri, M. (2018) Benchmarking Aggression Identification in Social Media. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC). pp. 1-11.

Malmasi, S., Zampieri, M. (2018) Challenges in Discriminating Profanity from Hate Speech. Journal of Experimental & Theoretical Artificial Intelligence. Volume 30, Issue 2, pp. 187-202. Taylor & Francis. 

Waseem, Z., Davidson, T., Warmsley, D. and Weber, I. (2017) Understanding Abuse: A Typology of Abusive Language Detection Subtasks. Proceedings of the Abusive Language Online Workshop.


Previous OffensEval

REPORT
Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N. and Kumar, R. (2019) SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation. pp. 75-86.

DATASET
Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N. and Kumar, R. (2019) Predicting the Type and Target of Offensive Posts in Social Media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 1415-1420.
Ċ
Marcos Zampieri,
6 Aug 2020, 21:59