Background and Motivation

Aggression and its manifestations in different forms have taken unprecedented proportions with the tremendous growth of Internet and social media. The research community, especially within the fields of Linguistics and Natural Language Processing, has responded by understanding the pragmatic and structural aspects of such forms of language usage ((Culpeper, 2011; Hardaker, 2010, 2013) and several other) and developing systems that could automatically detect and handle these ((Kumar et al., 2018; Zampieri et al., 2019; Waseem et al., 2017; Nobata et al., 2016; Dadvar et al., 2013) and numerous others).

In the ComMA project, we are working on these different aspects of aggressive and offensive language usage online and its automatic identification. As part of our efforts in the project, we present a novel multi-label classification task to the research community, in which each sample will be required to be classified as aggressive, gender biased or communally charged. We expect that the task will be interesting for researchers working in the different related areas of hate speech, offensive language, abusive language as well more generally in text classification.

Dataset

We are providing a multilingual dataset with a total of 12,000 samples for training and development and an overall 3,000 samples for testing in four Indian languages Meitei, Bangla (Indian variety), Hindi and English. The dataset is marked at three levels - aggression, gender bias and communal bias.

1. Sub-task A: Aggression Identification. The task will be to develop a classifier that could make a 3-way classification in between ‘Overtly Aggressive’(OAG), ‘Covertly Aggressive’(CAG) and ‘Non-aggressive’(NAG) text data.

2. Sub-task B: Gender Bias Identification. This task will be to develop a binary classifier for classifying the text as ‘gendered’(GEN) or ‘non-gendered’(NGEN).

3. Sub-task C: Communal Bias Identification: This task will be to develop a binary classifier for classifying the text as ‘communal' (COM) and 'non-communal'(NCOM).

The task could be approached as three separate classification tasks or a multi-label classification task or a structured classification task.

Some examples of possible annotations

• Aurat jaat is Kutiya jaat. Unfaithful jaat [Female species are bitch species. Unfaithful species] OAG GEN NCOM

• Homosexual brave nahi hote. [Homosexuals are not brave] NAG GEN NCOM

• General shaab sahi hain.inh logo ne azaadhi ka gala fayda uthaya hain in he SaudiArab,Dubai aur Malaysia ki tarah Korepadhne chaiye [General sahab is right. These people have misused freedom. They shouldbe flogged as in Saudi Arabu, Dubai andMalaysia] OAG NGEN COM

• Ye chasmis kutta hi he [This bespectacled person is a dog] OAG NGEN NCOM

Evaluation

The submissions will be evaluated and ranked on the basis of two standard evaluation metrics used for multi-label classification.

• instance-F1: It is the F-measure averaging on each instance in the test set i.e. the classification will be considered right only when all the labels in a given instance is predicted correctly. It will be the primary evaluation metric for the task.

• micro-F1: It gives a weighted average score of each class and is generally considered a good metric in cases of class-imbalance.

Instance-F1 gives an indication of the overall performance of the system while micro-F1 accounts for the partially correct predictions as well. Taken together they give an accurate evaluation of the classifier.