Trolling, Aggression and Cyberbullying

Please Click Here to Get the Dataset

[Licensed under Creative Common Non-Commercial Share-Alike 4.0 licence CC-BY-NC-SA 4.0]

Citations

If you are using the dataset and / or shared task report, kindly cite the following -

TRAC - 2 Shared Task Dataset

@InProceedings{trac2-dataset,

  author    = {Bhattacharya, Shiladitya  and  Singh, Siddharth  and  Kumar, Ritesh  and  Bansal, Akanksha  and  Bhagat, Akash  and  Dawer, Yogesh  and  Lahiri, Bornini  and  Ojha, Atul Kr.},

  title     = {Developing a Multilingual Annotated Corpus of Misogyny and Aggression},

  booktitle      = {Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying},

  month          = {May},

  year           = {2020},

  address        = {Marseille, France},

  publisher      = {European Language Resources Association (ELRA)},

  pages     = {158--168},

  url       = {https://www.aclweb.org/anthology/2020.trac2-1.25}

}

TRAC - Shared Task Report:

@InProceedings{trac2-report,

  author    = {Kumar, Ritesh  and  Ojha, Atul Kr.  and  Malmasi, Shervin  and  Zampieri, Marcos},

  title     = {Evaluating Aggression Identification in Social Media},

  booktitle      = {Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying},

  month          = {May},

  year           = {2020},

  address        = {Marseille, France},

  publisher      = {European Language Resources Association (ELRA)},

  pages     = {1--5},

 url       = {https://www.aclweb.org/anthology/2020.trac2-1.1}

}

The workshop includes two shared tasks as detailed below -

1. Sub-task A: Aggression Identification Shared Task. The task will be to develop a classifier that could make a 3-way classification in between ‘Overtly Aggressive’, ‘Covertly Aggressive’ and ‘Non-aggressive’ text data. We are making available a dataset of 5,000 aggression-annotated data from social media each in Bangla (in both Roman and Bangla script), Hindi (in both Roman and Devanagari script) and English for training and validation. We will release additional data for testing your system. The train and test sets for the tasks are different from the ones made available during TRAC - 1.
2. Sub-task B: Misogynistic Aggression Identification Shared Task: This task will be to develop a binary classifier for classifying the text as ‘gendered’ or ‘non-gendered’. We will provide a dataset of 5,000 annotated data from social media each in Bangla (in both Roman and Bangla script), Hindi (in both Roman and Devanagari script) and English for training and validation. We will release additional data for testing your system.

General Instructions for Participants

• Each team is allowed to submit up to three systems (each task) for evaluation.
• The test data will be sent to the participants on the 5th of March, 2020 and they will be given a window of 72 hours (i.e. till 8th of March, 2020 for testing your system and sending us back the labels for the test instances. We will send the participants further instructions on submitting your system and labels for the test data in due course of time.
• We expect each team to submit a system description paper after the evaluation. The deadline, length of submission and other instructions for the system description papers will be same as that for the workshop papers. All the system papers will be published in the proceedings and the best systems will be given slots for demos and presentations at the workshop.
• Participants can use additional data for training the system. Just make sure that the dataset that you use is either already publicly available or you make it available immediately after submission (and well before the submission of your system paper) and you mention it in your submission. Use of non-public additional data for training will disqualify your system.

Evaluation Metric

The submitted system will be evaluated on the basis of weighted macro-averaged F-scores. The individual F-score of each class will be weighted by the proportion of the concerned class in the test set and the final F-score will be the average of these individual F-scores of each class.

Dates

Training set release January 25, 2020

Test set release March 5, 2020

Submissions due March 8, 2020

Results announcement March 11, 2020

System papers deadline March 21, 2020

Reviews for papers March 31, 2020

Camera-ready versions April 02, 2020

[Timezone: as long as it’s the date mentioned, anywhere on earth; UTC-12.]

Dataset is now publicly available under Creative Commons Non-Commercial Share-Alike 4.0 licence CC-BY-NC-SA 4.0!