or email us at: semeval2022-iSarcasmEval-organizers@googlegroups.com
Sarcasm is a form of verbal irony that occurs when there is a discrepancy between the literal and intended meanings of an utterance. Through this discrepancy, the speaker expresses their position towards a prior proposition, often in the form of surface contempt or derogation (Wilson, 2006).
Sarcasm is omnipresent on the social web and, due to its nature, can be highly disruptive of computational systems that harness this data to perform tasks such as sentiment analysis, opinion mining, author profiling, and harassment detection (Liu,2012; Rosenthal et al., 2014; Maynard and Green-wood, 2014; Van Hee et al., 2018). In the context of SemEval, in particular, Rosenthal et al. (2014) show a significant drop in sentiment polarity classification performance when processing sarcastic tweets, compared to non-sarcastic ones. Such computational systems are widely deployed in industry, driving marketing, administration, and investment decisions (Medhat et al., 2014). As such, it is imperative to devise models for sarcasm detection.
Most previous textual sarcasm detection dataset shave been annotated using a weak supervision method. In this approach, texts are considered sarcastic if they meet predefined criteria, such as including specific tags (e.g. #sarcasm, #irony) (Ptacek et al., 2014; Khodak et al., 2018). However, this can lead to noisy labels for several reasons, as outlined by Oprea et. al (2020). Other work has relied on manual labelling, where sarcasm labels are provided by human annotators (Filatova, 2012; Riloff et al., 2013a; Abercrombie and Hovy, 2016). However, such labels represent annotator perception, which may differ from author intention, as further pointed out by Oprea et al. (2020).
Further, the vast majority of sarcasm detection work (Campbell and Katz, 2012; Riloff et al., 2013; Joshi et al., 2016; Wallace et al., 2015; Rajadesingan et al., 2015; Bamman and Smith, 2015;Amir et al., 2016; Hazarika et al., 2018; Oprea and Magdy, 2019) has focused exclusively on the English language and, due to the sociocultural aspects of sarcastic communication (Oprea and Magdy, 2020), it is unclear if models trained on English could generalise to other languages. To our knowledge, the small amount of work on other languages such as Arabic (Karoui et al., 2017; Ghanem et al.,2019; Abbes et al., 2020; Abu-Farha and Magdy, 2020) relies on the two labelling methods above.
Previous shared tasks on sarcasm detection (Van Hee et al., 2018; Ghanem et al., 2019; Ghosh and Muresan, 2020; Abu Farha et al., 2021) present datasets annotated via the two methods discussed above. The potential noisy labels that these methods can produce gives us reason to be concerned about the effectiveness of models that were trained on such datasets.
We introduce a new data collection method where the sarcasm labels for texts are provided by the authors themselves, thus eliminating labelling proxies (in the form of predefined tags, or third-party annotators).
We use this method to collect two datasets, one in English and one in Arabic.
Within each dataset, for each sarcastic text, we also ask its author to rephrase the text to convey the same intended message without using sarcasm.
Finally, we ask linguistic experts to further label each text into one of the categories of ironic speech defined by Leggitt and Gibbs (2000): sarcasm, irony, satire,understatement, overstatement, and rhetorical question.
For the Arabic dataset, we also include the dialect label of the text.
As such, each text in the datasets has the following information attached to it:
a label specifying its sarcastic nature (sarcastic or non-sarcastic), provided by its author;
a rephrase provided by its author that conveys the same message non-sarcastically;
a label specifying the category of ironic speech that it reflects, provided by a linguistic expert (English only); and
a label specifying the dialect (Arabic only).
The training data is available to download using the following links:
English data:
Arabic data: link
The complete dataset is available through this link
Using these two datasets we formulate three sub-tasks for each language:
SubTask A: Given a text, determine whether it is sarcastic or non-sarcastic;
SubTask B (English only): A binary multi-label classification task. Given a text, determine which ironic speech category it belongs to, if any;
SubTask C: Given a sarcastic text and its non-sarcastic rephrase, i.e. two texts that convey the same meaning, determine which is the sarcastic one.
Metrics:
For all the sub-tasks, precision, recall, accuracy and macro-F1 will be reported. The main metrics are:
SubTask A: F1-score for the sarcastic class.
SubTask B: Macro-F1 score
SubTask C: Accuracy
CODALAB available here
Your paper must use the official ACL style template. You can find the templates for LaTeX and World at: https://github.com/acl-org/acl-style-files
Please see the Paper Submission Requirements (https://semeval.github.io/paper-requirements.html) for more details on what is required.
Please see the Guidelines for SemEval System Papers (https://semeval.github.io/system-paper-template.html) for an overview of what to include in your papers (including examples).
System description papers for teams who participated in one Subtask must be no more than 5 pages long.
System description papers for teams who participated in more than one Subtask must be no more than 8 pages long.
References, Appendices and Acknowledgments are NOT counted in calculating page limits.
You will be provided with one additional page to incorporate reviewers suggestions.
For more information on papers and the workshop see the FAQ (https://semeval.github.io/faq.html)
Training set release 26/11/2021
Test set release. 18 Jan 2022
Submission deadline. 31 Jan 2022
Paper submission deadline. 23 Feb 2022 28 Feb 2022
Please join our Google group so you get the latest updates.
Ibrahim Abu Farha
The University of Edinburgh
Edinburgh, UK
Silviu Vlad Oprea
The University of Edinburgh
Edinburgh, UK
Steven Wilson
Oakland University
Rochester, MI, USA
Walid Magdy
The University of Edinburgh
Edinburgh, UK