Dialogue Breakdown Detection Challenge 4
About the challenge
Following the success of DBDC3, we are please to organize DBDC4 as part of the WOCHAT workshop held in Sicily in conjunction with IWSDS2019.
The task of dialogue breakdown detection is to detect whether the system utterance causes dialogue breakdown (a situation in a dialogue where users cannot proceed with the conversation) in a given dialogue context. The participants of this challenge will develop a dialogue breakdown detector that outputs a dialogue breakdown label (B: breakdown, PB: possible breakdown, or NB:Not a breakdown) and a distribution of these labels. Although we acknowledge that the schedule can be rather tight, we consider DBDC4 to be a perfect opportunity for you to test your dialogue models, especially in terms of contextual understanding and natural human-computer communication.
Task setting of dialogue breakdown detection
Dialogue example annotated with dialogue breakdown labels.
Schedule
- January 29: Announcement of the challenge
- Jan 29-March
2831: Registration period- Registration can be done at: https://goo.gl/forms/p0av6p6k8d3D0oWB3
March 29-April 4 April 1-April 4: Formal run
- Unlabeled test data released on
March 29April 1. - Participants submit results as well as draft system description papers by April 4.
- April 6: Evaluation results released and acceptance notification for system description papers
- All accepted system description papers will be presented as posters. The papers will be available online on the WOCHAT website. In addition, high-quality submissions will be considered for inclusion in the post-IWSDS 2019 contributed book.
- April 20: System description papers deadline for online proceedings
- April 24-26: DBDC4 (the session will be on April 24
on one of the days of IWSDS2019)
Dataset
Registered participants will have access to training and dev data. Evaluation data (unlabeld test data) will be distributed just before the formal run to the registered participants. We will distribute both English and Japanese datasets. A registered participant can submit up to five runs for each language. Prospective registrant can have a look at the datasets of DBDC3, which can be found here.
Getting started
You can have a look at this page to get started on dialogue breakdown detection challenge! This page shows how to use the baseline and evaluate dialogue breakdown detection results.
Sponsors
We are currently looking for sponsors, who will be able to talk during our special session at IWSDS 2019. In addition, sponsors will be credited in all datasets we distribute and advertised in our WOCHAT website.
Organizers
- Ryuichiro Higashinaka (NTT)
- Luis F. D'Haro (Technical University of Madrid)
- Bayan Abu Shawar (Arab Open University)
- Rafael Banchs (NTU)
- Kotaro Funakoshi (Kyoto University)
- Michimasa Inaba (Hiroshima City University)
- Yuiko Tsunomori (NTT Docomo)
- Tetsuro Takahashi (Fujitsu Laboratories, LTD)
- João Sedoc (University of Pennsylvania)
Contact us at: dbdc4-admin<at>googlegroups.com
References
- Higashinaka, Ryuichiro, Kotaro Funakoshi, Yuka Kobayashi, and Michimasa Inaba. "The dialogue breakdown detection challenge: Task description, datasets, and evaluation metrics." In LREC. 2016.
- Higashinaka, Ryuichiro, Kotaro Funakoshi, Michimasa Inaba, Yuiko Tsunomori, Tetsuro Takahashi, and Nobuhiro Kaji. "Overview of dialogue breakdown detection challenge 3." Proceedings of Dialog System Technology Challenge 6 (2017).
- Hori, Chiori, Julien Perez, Ryuichiro Higashinaka, Takaaki Hori, Y-Lan Boureau, Michimasa Inaba, Yuiko Tsunomori, Tetsuro Takahashi, Koichiro Yoshino, and Seokhwan Kim. "Overview of the sixth dialog system technology challenge: DSTC6." Computer Speech & Language 55 (2019): 1-25.
- Yuiko Tsunomori, Ryuichiro Higashinaka, Tetsuro Takahashi and Michimasa Inaba. "Evaluating dialogue breakdown detection in chat-oriented dialogue systems", In SEMDIAL, 2018.
- DBDC3 website: https://dbd-challenge.github.io/dbdc3/
- You can find datasets used in DBDC3 here: https://dbd-challenge.github.io/dbdc3/data/
- DSTC6 website: http://workshop.colips.org/dstc6/papers.html (papers of DBDC3 can be found on this website)