CRAC 2022

Massimo Poesio, Lori Levin: Annotating anaphoric reference in dialogue: the CODI/CRAC 2002 Shared Task corpus

(joint work with Maris Camilleri, Paloma Carretero Garcia, Taiqi He, Mark-Cristoph Mueller, Carolyn Rose, Michael Strube, Juntao Yu and Katherine Zhang)

Most current research on anaphoric reference focuses on news text, in particular written, and on identity anaphora (coreference). This is largely due to the lack of annotated datasets of a sufficient size to train and evaluate models for other genres, and other types of anaphoric reference. Arguably the most important among the understudied genres is conversational language in dialogue. Anaphora resolution in dialogue requires systems to handle grammatically incorrect language suffering from disfluencies and mentions jointly created across utterances (Poesio & Rieser, 2010) or whose function is to establish common ground rather than refer (Clark & Brennan, 1990; Heeman & Hirst, 1995). Dialogue involves much more deictic reference, vaguer anaphoric and discourse deictic reference, speaker grounding of pronouns and long-distance conversation structure. These complexities are normally absent from news or Wikipedia articles, which constitute the bulk of current datasets for coreference resolution (Poesio et al., to appear).

The series of CODI/CRAC Shared Tasks in Anaphora Resolution in Dialogue (Khosla et al., 2021; Yu et al., 2022) was organized to address this issue by creating datasets that our community could use to study anaphoric reference in different types of conversational setups, and to tackle less studied forms of anaphoric reference such as bridging reference or discourse deixis. The annotated corpus created for the CODI/CRAC series consists of conversations from four well-known conversational datasets: the AMI corpus (Carletta, 2006), the LIGHT corpus (Urbanek et al., 2019), the PERSUASION corpus (Wang et al., 2019) and SWITCHBOARD (Godfrey et al., 1992). These documents were annotated according to the annotation scheme for the ARRAU 3 corpus, which includes guidelines for identifying discontinuous markables and annotating split antecedent plurals, bridging reference, and discourse deixis. For this second edition, we created new test sets, but also systematically checked the data annotated for the first edition. As this annotation effort also involved annotators that had not been previously involved in the ARRAU 3 annotation, this work also involved extensive discussions about the scheme; new reliability tests of the annotation scheme were carried out, and the annotation guidelines were substantially revised.

Juntao Yu, Michal Novák: The recent developments in Universal Anaphora Scorer

The Universal Anaphora initiative aims to push forward the state of the art in anaphora and anaphora resolution by expanding the aspects of anaphoric interpretation which are or can be reliably annotated in anaphoric corpora, producing unified standards to annotate and encode these annotations, deliver datasets encoded according to these standards, and developing methods for evaluating models carrying out this type of interpretation. Such expansion of the scope of anaphora resolution requires a comparable expansion of the scope of the scorers used to evaluate this work. Last year, we introduce an extended version of the Reference Coreference Scorer (Pradhan et al., 2014) that can be used to evaluate identity anaphora resolution (including singletons, split-antecedents), bridging reference resolution, non-referring expressions and discourse deixis. The scorer has been used in the two recent CODI-CRAC Shared Tasks on Anaphora Resolution in Dialogues. Recently, an extension of the UA scorer that supports also discontinuous markables has been used by Novák et al (2022) in the CRAC 2022 Shared Task on Multilingual Coreference Resolution. In this talk, we will introduce the details about the scorer on scoring the different aspects of anaphora resolutions and how has it been used in recently shared tasks. In addition, we will also discuss the work in progress for the scorers such as mention overlap ratio, anaphor-decomposable score and the adaptation for CRAFT shared task.