Overview of the Task

Background and Motivation

Online gender-based violence is a growing challenge that compounds existing social and economic vulnerabilities. It can cause people to recede from online spaces impacting their political and economic opportunity. At its worst, it can lead to loss of life. While there is a need for automated approaches to detect gendered abuse, there is a lack of Indic language datasets that enable such approaches for Indian language content.

More information on how the dataset is created and its details, kindly refer to the dataset paper provided below. 

Link to the Dataset Paper - https://arxiv.org/abs/2311.09086

Dataset

We provide a novel dataset on gendered abuse in Hindi, Tamil and Indian English. his dataset was annotated by eighteen activists and researchers who have faced or studied gendered abuse. The dataset contains a total of 7638 posts in English, 7714 posts in Hindi, and 7914 posts in Tamil. We will provide a train and a test split for the dataset. 

The dataset has been annotated by asking 3 different questions(labels) to annotators. The dataset contains posts tagged with the following three labels:

1. Label 1 - Is this post gendered abuse when not directed at a person of marginalized gender and sexuality?

2. Label 2 - Is the post gendered abuse when directed at a person of marginalized gender and sexuality?

3. Label 3 - Is this post explicit/aggressive?

Some examples of possible annotations

The values for each label could be the following

1. "1" - Indicates the annotator believes the post (tweet) matches the label.

2. "0" - Indicates the annotator doesn’t believes the post (tweet) does not match the label.

3. "NL" - The post was assigned to the annotator but not annotated.

4. "NaN" - Indicates the post was not assigned to the annotator

Task Description

The proposed task is to develop a gendered abuse detection model based on the label(s) in the dataset. We propose the following three subtasks:-

1. Build a classifier using the provided dataset only to detect gendered abuse (label 1)

2. Use transfer learning from other open datasets for hatespeech and toxic language detection in Indic languages to build a classifier to detect gendered abuse (label 1)

3. Build a multi-task classifier that jointly predicts both gendered abuse (label 1) and explicit language (label 3)

Evaluation

The submissions will be evaluated and ranked on the basis of standard evaluation metrics used for multi-label classification.

• F-1 Score - It presents a good balance between precision and recall and gives good results on imbalanced classification problems.