This year MT track features two tasks, namely the Talk task and the Microsoft Speech Language Translation (MSLT) task.
The Talk task consists in subtitling videos of TED talks, as proposed in previous editions, and of QED lectures.
The TED exercise relies on TED talks, a collection of public speeches on a variety of topics for which video, transcripts and translations are available on the Web. The goal of this task is to translate TED subtitles, provided as segmented text. Training data is limited to a supplied collection of freely available parallel texts (see the information at the homepage), including a parallel corpus of TED Talks (see below the "In-Domain training and development data" entry).
The QED exercise asks participants to subtitle some educational videos and lectures from the QCRI Educational Domain (QED) Corpus. For this edition, it is considered as a dry-run and as such no development set will be released.
The goal of the MSLT task is to translate the manual transcripts of Skype calls from one language into another. In order to receive the development and/or evaluation data for the MSLT test task, participants need to sign this data license agreement and need to sign up in the participant registration form for the respective tasks. Please return your signed copy electronically to sebastian.stueker@kit.edu or fax it to +49 721 607 721. The data will then be made available to you.
Language directions:
Talk: English <-> {French, German, Czech, Arabic}
MSLT task: English -> {French, German}, German->English
Note on Arabic: the Arabic text needs to be normalized before scoring. We will process the references and submitted runs by means of the QCRI Arabic Normalizer 3.0 that can be downloaded from
http://alt.qcri.org/tools/arabic-normalizer/
Our installation is equipped with the following required and publicly available support tools:
MADA: Morphological Analysis and Disambiguation for Arabic, version 3.2
Aramorph, version 1.2.1: morphological analyzer for Arabic
SRILM, version 1.5.10 (disambig)
SVMTool, version 1.3.1 (SVMTagger)
In-Domain training and development data:
TED exercise: archives for each language pair can be downloaded from the WIT3 website
QED exercise: no development set is provided
MSLT task: development sets are available; for receiving and using them the aforementioned license agreement must be signed and sent to organizers. For training purposes, no specific data is released, but any linguistic resource listed here can be used.
Evaluation data:
Input format: NIST XML format, case sensitive source text with punctuation
Output format: NIST XML format, detokenized case sensitive translations with punctuation. NIST XML format is described in this paper (Section 8 and Appendix A); you can refer to these XML templates
Submission: please, refer to the submission guidelines provided below
text encoding: UTF8
MSLT task: tst2016 has been sent to participants who registered for the task and provided the organizers with the aforementioned license agreement properly signed.
Evaluation:
case sensitive BLEU and NIST scores are computed with the NIST script mteval-v13a.pl, while case sensitive TER score with tercom.7.25.jar. The respective invocations are:
mteval-v13a.pl -c
java -Dfile.encoding=UTF8 -jar tercom.7.25.jar -N -s
the internal tokenization of the two scorers is exploited
Evaluation Server:
An online Evaluation Server is available for MT tasks. Currently, it is enabled to score development sets; after Oct 14 (the end of the evaluation period of the MT track), it will score evaluation sets as well. There are two ways to use it:
via Web page
via REST Web service
Its use is restricted to the Participants. If you are interested in using it, please send an e-mail to cettolo@fbk.eu
Submission Guidelines:
Each participant has to submit at least one run for each translation task s/he registered for.
Detokenized case sensitive automatic translations with punctuation have to be wrapped in NIST XML formatted files. NIST XML format is described in this paper (Section 8 and Appendix A); you can refer to these XML templates
XML files with runs have to be submitted as a gzipped TAR archive (in the format specified below) and e-mailed to cettolo@fbk.eu and jan.niehues@kit.edu and sebastian.stueker@kit.edu
TAR archive file structure:
<UserID>/<Set>.<Task>.<UserID>.primary.xml
/<Set>.<Task>.<UserID>.contrastive1.xml
/<Set>.<Task>.<UserID>.contrastive2.xml
/...
where:
<UserID> = user ID (short name) of participant provided in the Registration Form
<Set> = IWSLT16.TED.tst2016 | IWSLT16.TED.tst2015 | IWSLT16.QED.tst2016 | IWSLT16.MSLT.tst2016
<Task> = MT_<fromLID>-<toLID>
<fromLID>, <toLID> = Language identifiers (LIDs) as given by ISO 639-1 codes; see here examples of language codes.
The PRIMARY run for each task will be used for the official scoring; nevertheless, CONTRASTIVE runs will be evaluated as well. In the same archive, runs for different tasks (i.e. language pairs) can be included.
Example:
fbk/IWSLT16.TED.tst2015.MT_en-fr.fbk.primary.xml
/IWSLT16.TED.tst2016.MT_en-fr.fbk.primary.xml
/IWSLT16.TED.tst2016.MT_en-fr.fbk.contrastive1.xml
/IWSLT16.TED.tst2016.MT_en-fr.fbk.contrastive2.xml
/IWSLT16.TED.tst2015.MT_cs-en.fbk.primary.xml
/IWSLT16.TED.tst2016.MT_cs-en.fbk.primary.xml
Re-submitting runs is allowed as far as the mails arrive BEFORE the submission deadline. In case that multiple TAR archives are submitted by the same participant, only runs of the most recent submission mail will be used for the IWSLT 2016 evaluation and previous mails will be ignored.