Dialogues Task

The Dialogues task addresses the text translation of dialogues, from Japanese to English.

The dialogues were between an elderly person and listener for information navigation and attentive listening, and transcribed manually.

The dialogue corpus was developed in Nara Institute of Science and Technology (NAIST) and distributed publicly for research purposes,

and English translation of the dialogues are provided by National Institute of Information and Communications Technology (NICT) for this IWSLT evaluation campaign.

In-domain datasets will be provided only for development and test.

One important challenge in this task will be in considering dialogue contexts because utterances in spoken dialogues have to be understood with their contexts such as pronouns and empty categories.

• Training data:

  • In-domain training data are not available.

  • Participants can use any publicly available Japanese-English bilingual corpora for this task, not just listed below. Used resources have to be mentioned in system description papers.

  • Some Japanese-English parallel corpora are available (under their terms of use):

    • WIT³ (TED Talks) https://wit3.fbk.eu/mt.php?release=2017-01-trnted

    • Japanese-English Bilingual Corpus of Wikipedia's Kyoto Articles http://alaginrc.nict.go.jp/WikiCorpus/index_E.html

      • Pre-processed corpus avaiable as Kyoto Free Translation Task Data http://www.phontron.com/kftt/

    • Tatoeba https://tatoeba.org/eng/downloads

    • WordNet JE http://compling.hss.ntu.edu.sg/wnja/index.en.html

    • JEC basic sentences http://nlp.ist.i.kyoto-u.ac.jp/index.php?%E6%97%A5%E8%8B%B1%E4%B8%AD%E5%9F%BA%E6%9C%AC%E6%96%87%E3%83%87%E3%83%BC%E3%82%BF

• Development Data:

  • In-domain development set is now available. Click to download (revision 2.0: license included; 2018/4/27)

• Evaluation data:

  • test set will be released at the beginning of the evaluation period.

  • Evaluation set is now available. Click to download (revision 2.0: license included; 2018/4/27)

• Evaluation process:

  • Case sensitive BLEU and NIST scores are computed with the NIST script mteval-v13a.pl, while case sensitive TER score with tercom.7.25.jar. The respective invocations are:

  • - mteval-v13a.pl -c

  • - java -Dfile.encoding=UTF8 -jar tercom.7.25.jar -N -s

  • The internal tokenization of the two scorers is exploited.

Submission Guidelines for Dialogues task

Each participant has to submit at least one run for each translation task s/he registered for.

Detokenized case sensitive automatic translations with punctuation have to be wrapped in NIST XML formatted files. NIST XML format is described in this paper (Section 8 and Appendix A); XML templates will be made available by the evaluation period.

XML files with runs have to be submitted as a gzipped TAR archive (in the format specified below) and e-mailed to sudohATisDOTnaistDOTjp

TAR archive file structure:

<UserID>/<Set>.<Task>.<UserID>.primary.xml

/<Set>.<Task>.<UserID>.contrastive1.xml

/<Set>.<Task>.<UserID>.contrastive2.xml

/...

where:

<UserID> = user ID (short name) of participant provided in the Registration Form

<Set> = IWSLT17.tst2017

<Task> = dialogues_ja-en

The PRIMARY run will be used for the official scoring; nevertheless, CONTRASTIVE runs will be evaluated as well.

Example:

naist/IWSLT17.tst2017.dialogues_ja-en.naist.primary.xml

/IWSLT17.tst2017.dialogues_ja-en.naist.contrasive1.xml

Re-submitting runs is allowed as far as the mails arrive BEFORE the submission deadline. In case that multiple TAR archives are submitted by the same participant, only runs of the most recent submission mail will be used for the IWSLT 2017 evaluation and previous mails will be ignored.