We had a very successful DBDC4 at IWSDS2019! We thank everyone who participated in the event!

The proceedings of DBDC4 can be found here!

The datasets are now available here!

About the challenge

Following the success of DBDC3, we are please to organize DBDC4 as part of the WOCHAT workshop held in Sicily in conjunction with IWSDS2019.

The task of dialogue breakdown detection is to detect whether the system utterance causes dialogue breakdown (a situation in a dialogue where users cannot proceed with the conversation) in a given dialogue context. The participants of this challenge will develop a dialogue breakdown detector that outputs a dialogue breakdown label (B: breakdown, PB: possible breakdown, or NB:Not a breakdown) and a distribution of these labels. Although we acknowledge that the schedule can be rather tight, we consider DBDC4 to be a perfect opportunity for you to test your dialogue models, especially in terms of contextual understanding and natural human-computer communication.

Schedule

• January 29: Announcement of the challenge
• Jan 29-March 28 31: Registration period
  • Registration can be done at: https://goo.gl/forms/p0av6p6k8d3D0oWB3

March 29-April 4 April 1-April 4: Formal run

• Unlabeled test data released on March 29 April 1.
• Participants submit results as well as draft system description papers by April 4.
• April 6: Evaluation results released and acceptance notification for system description papers
  • All accepted system description papers will be presented as posters. The papers will be available online on the WOCHAT website. In addition, high-quality submissions will be considered for inclusion in the post-IWSDS 2019 contributed book.
• April 20: System description papers deadline for online proceedings
• April 24-26: DBDC4 (the session will be on April 24 on one of the days of IWSDS2019)

Dataset

Registered participants will have access to training and dev data. Evaluation data (unlabeld test data) will be distributed just before the formal run to the registered participants. We will distribute both English and Japanese datasets. A registered participant can submit up to five runs for each language. Prospective registrant can have a look at the datasets of DBDC3, which can be found here.

Getting started

You can have a look at this page to get started on dialogue breakdown detection challenge! This page shows how to use the baseline and evaluate dialogue breakdown detection results.

Sponsors

We are currently looking for sponsors, who will be able to talk during our special session at IWSDS 2019. In addition, sponsors will be credited in all datasets we distribute and advertised in our WOCHAT website.

Organizers

• Ryuichiro Higashinaka (NTT)
• Luis F. D'Haro (Technical University of Madrid)
• Bayan Abu Shawar (Arab Open University)
• Rafael Banchs (NTU)
• Kotaro Funakoshi (Kyoto University)
• Michimasa Inaba (Hiroshima City University)
• Yuiko Tsunomori (NTT Docomo)
• Tetsuro Takahashi (Fujitsu Laboratories, LTD)
• João Sedoc (University of Pennsylvania)

Contact us at: dbdc4-admin<at>googlegroups.com

References

• Higashinaka, Ryuichiro, Kotaro Funakoshi, Yuka Kobayashi, and Michimasa Inaba. "The dialogue breakdown detection challenge: Task description, datasets, and evaluation metrics." In LREC. 2016.
• Higashinaka, Ryuichiro, Kotaro Funakoshi, Michimasa Inaba, Yuiko Tsunomori, Tetsuro Takahashi, and Nobuhiro Kaji. "Overview of dialogue breakdown detection challenge 3." Proceedings of Dialog System Technology Challenge 6 (2017).
• Hori, Chiori, Julien Perez, Ryuichiro Higashinaka, Takaaki Hori, Y-Lan Boureau, Michimasa Inaba, Yuiko Tsunomori, Tetsuro Takahashi, Koichiro Yoshino, and Seokhwan Kim. "Overview of the sixth dialog system technology challenge: DSTC6." Computer Speech & Language 55 (2019): 1-25.
• Yuiko Tsunomori, Ryuichiro Higashinaka, Tetsuro Takahashi and Michimasa Inaba. "Evaluating dialogue breakdown detection in chat-oriented dialogue systems", In SEMDIAL, 2018.
• DBDC3 website: https://dbd-challenge.github.io/dbdc3/
  • You can find datasets used in DBDC3 here: https://dbd-challenge.github.io/dbdc3/data/
• DSTC6 website: http://workshop.colips.org/dstc6/papers.html (papers of DBDC3 can be found on this website)