Discourse Relation Parsing and Treebanking (DISRPT)

Shared Task on Discourse Segmentation, Connective and Relation Identification across Formalisms

In conjunction with EMNLP 2021 and CODI 2021 workshop

November, 2021

News: Final results are now published! See the results here.

You can take a look at the data for the shared task in our repository: https://github.com/disrpt/sharedtask2021

Please check our FAQ page for more information about participation, evaluation etc.!

Google group for participants: disrpt2021_participants@googlegroups.com

Study of coherence relations in frameworks such as RST (Mann & Thompson 1988), SDRT (Asher & Lascarides 2003) and PDTB (Miltsakaki et al. 2004), has experienced a revival in the last few years, in English and many other languages (Matthiessen & Teruya 2015; Maziero et al. 2015; da Cunha 2016; Iruskieta et al. 2016; Zeldes 2016, 2017). Multiple sites are now actively engaged in the development of discourse parsers (Lin et al. 2014, Feng and Hirst 2014; Ji and Eisenstein 2014; Joty et al. 2015; Surdeanu et al. 2015; Xue et al. 2016; Braud et al. 2017; Guz & Carenini 2020), as a goal in itself, but also for applications such as sentiment analysis, argumentation mining, summarization, question answering, or machine translation evaluation (Benamara et al., 2017; Gerani et al. to appear; Durrett et al. 2016; Peldszus & Stede 2016; Scarton et al. 2016; Schouten & Frasincar 2016; among many others). At the same time, evaluation of results in discourse parsing has proven complicated (see Morey et al. 2017), and progress in integrating results across discourse treebanking frameworks has been slow. We propose shared tasks on discourse unit segmentation, connective identification and discourse relation classification across formalisms (see below) that aim to promote convergence of resources and a joint evaluation of discourse parsing approaches.

Description of the Tasks

The first two tasks were already proposed in 2019. You can consult the results here and the associated papers here.

TASK 1: Discourse Unit Segmentation across Formalisms

The DISRPT 2019 workshop introduced the first iteration of a cross-formalism shared task on discourse unit segmentation. Since all major discourse parsing frameworks imply a segmentation of texts into segments, learning segmentations for and from diverse resources is a promising area for converging methods and insights. We provide training, development, and test datasets from all available languages and treebanks in the RST, SDRT and PDTB formalisms, using a uniform format. Because different corpora, languages and frameworks use different guidelines for segmentation, the shared task is meant to promote design of flexible methods for dealing with various guidelines, and help to push forward the discussion of standards for discourse units. For datasets which have treebanks, we will evaluate in two different scenarios: with and without gold syntax, or otherwise using provided automatic parses for comparison.

TASK 2: Discourse Connective Identification across Languages

The DISRPT 2019 workshop also provided the first iteration of a shared task on cross-lingual connective identification, using data annotated within the PDTB framework. We repeat the task in 2021 with updated datasets aiming to identify the location of discourse connectives indicating the presence of coherence relations.

TASK 3: Discourse Relation Classification across Formalisms

We also propose the first iteration of a cross-formalism shared task on discourse relation classification. Data is converted from three distinct, but overlapping frameworks: Rhetorical Structure Theory (RST, Mann & Thompson 1988), Penn Discourse Treebank (PDTB, Miltsatsaki et al. 2004), and Segmented Discourse Representation Theory (SDRT, Asher & Lascarides 2003). The goal of the shared task is to bring together diverse strands of research on discourse relation identification, which are sometimes siloed separately due to differences between underlying data structures and assumptions of different frameworks. In order to enable approaches benefiting from multiple datasets created using distinct points of view, the task aims to find a common denominator in representing all available datasets, for the widest possible range of languages.

Shared Task Data and Formats

Data for the shared task will be soon released via GitHub together with format documentation and tools. For the 2019 shared task datasets, see https://github.com/disrpt/sharedtask2019.

Important dates

• March, 2021 - shared task sample data release

• May, 2021 - training data release

• 1 July 2021 – Test data release

• 5 July 2021 - Anonymity period begins

• 5 August 2021 – Submission of system and paper

• 6 September 2021 - Notification of acceptance

• 16 September 2021 - Camera-ready paper due

• November 11th - CODI-DISRPT

Google group for participants, please join us on: disrpt2021_participants@googlegroups.com to receive e-mail updates whenever new data is made available for the shared task.

To contact the organisers, please send an email to disrpt2021@googlegroups.com.

References

Asher, Nicholas, and Alex Lascarides. 2003. Logics of Conversation. Cambridge: Cambridge University Press.

Benamara, Farah, Maite Taboada & Yannick Mathieu. 2017. Evaluative language beyond bags of words: Linguistic insights and computational applications. Computational Linguistics 43(1), 201–264.

Braud, Chloé, Maximin Coavoux & Anders Søgaard. 2017. Cross-lingual RST discourse parsing. Proceedings of EACL 2017. Valencia, Spain, 292–304.

da Cunha, Iria. 2016. Towards discourse parsing in Spanish. Papers presented at TextLink - Structuring Discourse in Multilingual Europe - Second Action Conference. Budapest, Hungary.

Durrett, Greg, Taylor Berg-Kirkpatrick & Dan Klein. 2016. Learning-based single-document summarization with compression and anaphoricity constraints. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Berlin, Germany, 1998–2008.

Feng, Vanessa Wei & Graeme Hirst. 2014. A linear-time bottom-up discourse parser with constraints and post-editing. Proceedings of ACL 2014. Baltimore, MD, 511–521.

Gerani, Shima, Giuseppe Carenini & Raymond Ng. to appear. Modeling content and structure for abstractive review summarization. Computer Speech and Language.

Guz, Grigorii and Giuseppe Carenini. 2020. Corefer-ence for Discourse Parsing: A Neural Approach.In Proceedings of the First Workshop on Computa-tional Approaches to Discourse, pages 160–167, On-line. Association for Computational Linguistics.

Iruskieta, Mikel, Gorka Labaka & Juliano Desiderato Antonio. 2016. Detecting the central units in two different genres and languages: A preliminary study of Brazilian Portuguese and Basque texts. Procesamiento del Lenguaje Natural 56, 65–72.

Ji, Yangfeng and Jacob Eisenstein. 2014. Representation Learning for Text-Level Discourse Parsing. In Proceedings of ACL 2014, pages 13–24, Baltimore, MD.

Joty, Shafiq, Giuseppe Carenini & Raymond Ng. 2015. CODRA: A novel discriminative framework for rhetorical analysis. Computational Linguistics 41(3), 385–435.

Lin, Z., Ng, H., & Kan, M. (2014). A PDTB-Styled End-to-End Discourse Parser. Nat. Lang. Eng., 20, 151-184.

Mann, William C., and Sandra A. Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text-Interdisciplinary Journal for the Study of Discourse 8(3), 243–281.

Matthiessen, Christian M.I.M. & Kazuhiro Teruya. 2015. Grammatical realizations of rhetorical relations in different registers. Word 61(3), 232–281.

Maziero, Erick G., Graeme Hirst & Thiago A. S. Pardo. 2015. Semi-supervised never-ending learning in rhetorical relation identification. Proceedings of Recent Advances in Natural Language Processing, Hissar, Bulgaria.

Miltsakaki, Eleni, Rashmi Prasad, Aravind K. Joshi & Bonnie L. Webber. 2004. The Penn Discourse Treebank. In Proceedings of LREC 2004. Lisbon, Portugal.

Morey, Mathieu, Philippe Muller & Nicholas Asher. 2017. How Much Progress have we Made on RST Discourse Parsing? A Replication Study of Recent Results on the RST-DT. In: Proceedings of EMNLP 2017. Copenhagen, Denmark, 1319–1324.

Peldszus, Andreas & Manfred Stede. 2016. Rhetorical structure and argumentation structure in monologue text. Proceedings of the 3rd Workshop on Argument Mining, ACL. Berlin, Germany, 103–112.

Riccardi, Giuseppe, Frederic Bechet, Morena Danieli, Benoit Favre, Robert Gaizauskas, Udo Kruschwitz & Massimo Poesio. 2015. The SENSEI Project: Making sense of human conversations. In J. F. Quesada, F. J. Martín Mateos & T. López-Soto (eds.), Future and Emergent Trends in Language Technology. Proceedings of the First International FETLT Workshop. Berlin: Springer, 10–33.

Schouten, Kim & Flavius Frasincar. 2016. COMMIT at SemEval-2016 Task 5: Sentiment analysis with Rhetorical Structure Theory. Proceedings of SemEval-2016. San Diego, CA, 356–360.

Scarton, Carolina, Daniel Beck, Kashif Shah, Karin Sim Smith & Lucia Specia. 2016. Word embeddings and discourse information for Machine Translation Quality Estimation. Proceedings of the First Conference on Machine Translation, ACL. Berlin, Germany, 831–837.

Surdeanu, Mihai, Thomas Hicks & Marco Valenzuela-Escárcega. 2015. Two practical Rhetorical Structure Theory parsers. Proceedings of NAACL 2015. Denver, CO, 1–5.

Xue, Nianwen, Hwee Tou Ng, Sameer Pradhan, Attapol T. Rutherford, Bonnie Webber, Chuan Wang & Hongmin Wang. 2016. CoNLL 2016 Shared Task on multilingual shallow discourse parsing. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Berlin, Germany, 1–19.

Zeldes, Amir. 2016. rstWeb: A browser-based annotation interface for Rhetorical Structure Theory and discourse relations. Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2016) System Demonstrations. San Diego, CA, 1–5.

Zeldes, Amir. 2017. The GUM Corpus: Creating Multilayer Resources in the Classroom. Language Resources and Evaluation 51(3), 581–612.