IberAutextification 🤖👩🏻

 Welcome 🤗 

The IberAuTexTification: Automated Text Identification on Languages of the Iberian Peninsula shared task will take place as part of IberLEF 2024, the 6th Workshop on Iberian Languages Evaluation Forum at the SEPLN 2024 Conference, which will be held in Valladolid, Spain on the 26th of September, 2024.

IberAuTexTification is the second version of the AuTexTification at IberLEF 2023 shared task (Sarvazyan et al., 2023). From Genaios and the UPV, we extend the previous task in three dimensions: more models, more domains and more languages from the Iberian Peninsula (in a multilingual fashion), aiming to build more generalizable detectors and attributors. In this task, participants must develop models that exploit clues about linguistic form and meaning to identify automatically generated texts from a wide variety of models, domains, and languages. We plan to include LLMs like GPT-3.5, GPT-4, LLaMA, Coral, Command, Falcon, MPT, among others. New domains like essays, or dialogues, and cover the most prominent languages from the Iberian Peninsula: Spanish, Catalan, Basque, Galician, Portuguese, and English (in Gibraltar).  

A novelty from this edition is to detect in a multilingual (languages from the Iberian peninsula such as Spanish, English, Catalan, Gallego, Euskera, and Portuguese), multi-domain (news, reviews, emails, essays, dialogues, wikipedia, wikihow, tweets, emails, etc.), and multi-model (GPT, LLaMA, Mistral, Cohere, Anthropic, MPT, Falcon, etc.) setup, whether a text has been automatically generated or not, and, if generated, identify the model that generated the text. 

To foster engagement and reward dedication, we will award the best participant in each subtask with 500€ sponsored by Genaios. We hope for your participation and good luck in the competition! 🍀

To reduce your workload participating in the task, we provide a repository with baselines, evaluation code, and format checkers.The code is prepared with extensibility in mind, so you can use it as basis to develop your own models and get some functionalities for free as CLI endpoints, caching, or config handling. 

Introduction

The new generation of Large Language Models (LLMs) for text generation has surged and has been established in individual and company's workflows for many different tasks. These LLMs are trained to predict the next word on large-scale, curated datasets of Internet text and can now generate fluent, coherent, and plausible-looking text that creates the impression of greatness, not only in English, but in many other languages including those from the Iberian Peninsula. The most recent and prominent examples are the GPT family of language models developed by OpenAI, including GPT-3 (Brown et al., 2020), GPT-3.5 (Ouyang et al., 2022), and GPT4 (OpenAI, 2023); the LLaMA family from Meta (Touvron et al., 2023), and PaLM from Google (Google, 2023). Besides, many other companies are developing their own text generation solutions like Mistral (Jian et al., 2023), Cohere (Cohere, 2023), Anthropic (Anthropic, 2023), Falcon (Ebtesam et al., 2023), or Mosaic (Mosaic ML, 2023), while the research community is continually working with open-source base models to further improve their capabilities (Huggingface, 2023). 

Considering the current impact and popularity of these technologies to develop innovative applications, it is reasonable to think that there is certain amount of people that can use them to write machine-generated text (MGT) for malicious purposes such as disinformation campaigns to spread fake news or polarised opinions, increase the credibility of phishing campaigns, etc. Besides, the number of LLMs is continually growing, and most of them are available through pre-trained checkpoints in model hubs, or through free/under-subscription APIs, which increases the amount of potential malicious users. Malicious users can attack in many different languages, domains, models, or generation strategies, thus hardening the moderation. Therefore, it is necessary to develop effective and generalizable content moderation strategies to deal with MGT, based on (i) detecting whether a text has been generated by an LLM, and (ii) identify the model that generated a MGT for further forensic purposes.

There is a great interest in addressing these tasks (Deng et al., 2022), (Tourille et al., 2022), and they have special relevance for AI, NLP research hubs and companies that see MGT as an imminent threat to its reputation. Companies are highly interested in detecting automatically generated contents to foster/hinder the reputation of products and brands, and to verify the content at news or statements. This shared task aims to foster the R&D in generated text identification, to progress at the same pace as generation technologies do. The alternative scenario would result in different entities such as companies having difficulties dealing with generated and malicious content such as spam, fake opinions, and reviews. The new generation of opinion span techniques requires its corresponding new generation of counter-techniques to moderate them.  

 What has already been done?

It has been shown empirically that the automatic detection of MGT from specific models and domains can be performed with very high accuracy (Jawahar et al., 2020) (Bakhtin et al., 2019) (Sarvazyan et al., 2023). However, this is not realistic, since the combinatoric is large in the wild and it is required to develop generalizable detection and attribution models. In general, LLMs introduce statistical artifacts and common structures in their texts, that can be used as clues, either by humans or by discriminator systems, to identify automatically generated texts to some extent (Ippolito et al., 2022). For instance, some language models generate syntactic or grammatical artifacts, while most of them struggle with factual hallucinations and reasoning (Liu et al., 2022) due to their incapability of memorizing all the facts (Borgeaud et al., 2022) and outdated factual knowledge (Dhingra et al., 2022). Besides, the research community has found that existing detectors exhibit poor cross-domain accuracy, that is, they are not generalizable to different writing styles and publication formats, while accuracy in in-domain settings is near to perfect (Jawahar et al., 2020) (Bakhtin et al., 2019). 

 Can you spot generated text? 🤔

Could you spot whether the following texts have been automatically generated? Just consider that automatically generated text could show factual, grammatical, or coherence artifacts (Massarelli et al., 2020), along with statistical abnormalities that make the distributions of automatic and human texts differ (Ippolito et al., 2020), despite having well-shaped form (Bender et al., 2020).

 References

• Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9. 

• Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., & Lowe, R. (2022). Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. 

• Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., & Fiedel, N. (2022). Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. 

• Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., & Manica, M. (2022). Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. 

• Jawahar, G., Abdul-Mageed, M., & Lakshmanan, L. (2020). Automatic Detection of Machine Generated Text: A Critical Survey. In Proceedings of the 28th International Conference on Computational Linguistics (pp. 2296–2309). International Committee on Computational Linguistics.

• Deng, R., & Duzhin, F. (2022). Topological Data Analysis Helps to Improve Accuracy of Deep Learning Models for Fake News Detection Trained on Very Small Training Sets. Big Data and Cognitive Computing, 6(3). 

• Tourille, J., Sow, B., & Popescu, A. (2022). Automatic Detection of Bot-Generated Tweets. In Proceedings of the 1st International Workshop on Multimedia AI against Disinformation (pp. 44–51). Association for Computing Machinery. 

• Rodriguez, J., Hay, T., Gros, D., Shamsi, Z., & Srinivasan, R. (2022). Cross-Domain Detection of GPT-2-Generated Technical Text. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 1213–1233). Association for Computational Linguistics. 

• Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2020). Automatic Detection of Generated Text is Easiest when Humans are Fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 1808–1822). Association for Computational Linguistics.

• Uchendu, A., Le, T., Shu, K., & Lee, D. (2020, November). Authorship attribution for neural text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 8384-8395). 

• Clark, E.; August, T.; Serrano, S.; Haduong, N.; Gururangan, S.; and Smith, N. A. 2021. All That’s ‘Human’Is Not Gold: Evaluating Human Evaluation of Generated Text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 7282–7296. 

• Ethayarajh, K., & Jurafsky, D. (2022). How human is human evaluation? Improving the gold standard for NLG with utility theory. arXiv preprint arXiv:2205.11930. 

• Dugan, L., Ippolito, D., Kirubarajan, A., & Callison-Burch, C. (2020). RoFT: A Tool for Evaluating Human Detection of Machine-Generated Text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (pp. 189–196). Association for Computational Linguistics. 

• Massarelli, L., Petroni, F., Piktus, A., Ott, M., Rocktäschel, T., Plachouras, V., Silvestri, F., & Riedel, S. (2020). How Decoding Strategies Affect the Verifiability of Generated Text. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 223–235). Association for Computational Linguistics.

• Bender, E., & Koller, A. (2020). Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 5185–5198). Association for Computational Linguistics.