Embedded Files
Description of the task
Inferring ironic meanings is an easy task to humans, yet some of the speech acts involved in this operation might still cause communicational misunderstandings [1]. To create methods to automatically understand ironic text can be a challenging task, but it is crucial to improve performance of other NLP’s tasks, e.g. Sentiment Analysis [2] and Hate Speech Detection [3].
This task aims to instigate participants applying their solutions for Irony Detection in Portuguese. The availability of corpora written in Portuguese is scarce, which limits the amount of research done for this language.
This task will contribute to the progress of Portuguese NLP, as there is a demand in the area for the development of new methods and tools. Previous irony detection competitions, such as IDAT [4], IroSvA, [5], IronITA 2018 [6], and SemEval 2017 Task 4 [2], inspired us to develop a specific task for Portuguese.
Corpora
The corpora contains texts (tweets and news) about different topics written in Portuguese. In this task, we used corpora developed previously by [7], [8], and [1].
Training data will be drawn of publicly datasets of tweets (https://github.com/fabio-ricardo/deteccao-ironia) and news (https://github.com/schuberty/PLNCrawler).
Test dataset will contain a new set of tweets and news manually annotated by the organizers.
The two types of data (tweets and news) are evaluated separately.
Evaluation measures
Participating teams will be provided with training and test datasets. The metrics (Accuracy, Precision, Recall, F1, and Bacc) will be used for assessing the performance of the participating systems. The five measures will be calculated per class label. The submissions will be ranked according to Bacc, which avoids inflated performance estimates on imbalanced datasets.
Target audience
The expected target is anyone interested in figurative language / irony processing or enhancing the performances of other NLP applications, like Sentiment Analysis or Hate Speech Detection. We hope for substantial engagement of researchers, students, industrial teams, and practitioners of private companies.
References
L. A. de Freitas, A. Vanin, D. Hogetop, M. Bochernitsan, R. Vieira. Pathways for irony detection in tweets. In Proceedings of the 29th Symposium On Applied Computing (SAC-2014), pages 628-633, Gyeongju, Korea, 2014. ACM.
R. K. Gupta and Y. Yang. CrystalNest at SemEval-2017 Task 4 : Using Sarcasm Detection for Enhancing Sentiment Classification and Quantification. In Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 626-633, Vancouver, Canada, 2017. Association for Computational Linguistics.
C. Bosco, D. O. Felice, F. Poletto, M. Sanguinetti, T. Maurizio. Overview of the EVALITA 2018 hate speech detection task. In Proceedings of the 6th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA-2018), Turin, Italy, 2018. CEUR.org.
B. Ghanem, J. Karoui, F. Benamara, V. Moriceau, P. Rosso. IDAT@FIRE2019: Overview of the Track on Irony Detection in Arabic Tweets. In Proceedings of the Forum for Information Retrieval Evaluation (FIRE-2019), pages 380-390, Kolkata, India, 2019. CEUR.org.
R. Ortega-Bueno, F. Rangel, D. I. H. Farías, P. Rosso, M. Montes-y-Gómez, J. Medina-Pagola. Overview of the Task on Irony Detection in Spanish Variants (IroSvA). In Proceedings of the Iberian Languages Evaluation Forum (Iberlef-20019) co-located with 35th Conference of the Spanish Society for Natural Language Processing (SEPLN 2019), pages 229-256, Bilbao, Spain, 2019. CEUR.org.
F. R. A. da Silva. Detecção de Ironia e Sarcasmo em Língua Portuguesa: uma abordagem utilizando Deep Learning. 2018. Bachelor of Computer Science - Universidade Federal do Mato Grosso, Cuiabá, 2018.
G. Schubert, L. A. de Freitas. The Construction of a Corpus for Detecting Irony and Sarcasm in Portuguese. In Proceedings of XVII Encontro Nacional de Inteligência Artificial e Computacional (ENIAC-2020), pages 709-717, Rio Grande, Brazil, 2020. SBC.org.br.
Page updated
Report abuse