Shared task on Dialogue Natural Language Inference

Note: This is preliminary information. More concrete information, instructions and a CfP will be announced soon.

Introduction

In this shared task, participants design a system that can determine the relationship between a sequence of turns and a statement about the dialogue state. The system is presented with a fragment of dialogue, followed by a statement which is to be classified as either “entailment”, “contradiction” or “neutral” in relation to the dialogue state so far. Then, the next fragment of the dialogue is presented, followed by another statement to be classified.

Data

The training data consists of dialogues extracted from the CHILDES corpus (Warren-Leubecker, A. (1982) and Warren-Leubecker, A., & Bohannon, J. N. (1984), Bliss, L. (1988).), BNC 2014 (Love, R., Dembry, C., Hardie, A., Brezina, V. and McEnery, T. (2017)), and possibly additional corpora. The data is formatted as follows:

Speaker: Utterance

…

Speaker: Utterance

[CLASS] statement

Speaker: Utterance

…

Speaker: Utterance

[CLASS] statement

…

The goal of this task is to develop a system that can differentiate between statements that stand in an entailment, contradictory, or neutral relationship to the dialogue state. We will release data for training, development and testing. The training and development data will contain the true human-annotated class of each statement. Participants using statistical or deep learning models are allowed to use the additional data of their choosing.

Evaluation criteria

The systems will be evaluated using F1-score and accuracy.

Timeline

• Training and development data released

• Test data released

• Submit results

• Results announced

• Submission of system description papers

• Notification of acceptance

• Camera ready papers due

• Workshop

Organizers

• Adam Ek, CLASP, University of Gothenburg

• William Noble, CLASP, University of Gothenburg

• Stergios Chatzikyriakidis, CLASP, University of Gothenburg

Submission