Submission

Evaluation plan  

MERLIon CCS Challenge Evaluation Plan v1.2 (Updated 17th Feb 2023)

MERLIon CCS Challenge Evaluation Plan v1.1 

MERLIon CCS Challenge Evaluation Plan v1.0


Results submission

For result submission and scoring, we will be using CodaLab.

Account creation on CodaLab

Daily and lifetime submission limits are tied to user accounts, so it is imperative that each team use a SINGLE account to make ALL submissions. 


Troubleshooting 

Setting Up Your Team Name

Registering for Tracks on CodaLab

Each track and task combination in the challenge will be assigned a separate competition page on CodaLab. 

The pages are:

Before submitting to a track, you will have to register in the track. To register, navigate to the competition page of the track, click on the Participate tab, accept the terms and conditions, and click Register. A member of our team will then review your registration request and approve it. Upon acceptance you will receive an email titled “Accepted into MERLION Challenge...”. 

IMPORTANT: Your CodaLab account MUST use the same email address that you provided during registration for the MERLION Challenge. If the addresses differ, your request will be denied. 

Formatting Your Submission

Results Zip Archive Format for Task 1

For Task 1 (Language Identification), prediction results for all audio segments to be evaluated should be enclosed in a single .txt file. All audio segments labelled as English and Mandarin and do not overlap with another language must be included. The order of the audio segments must be according to the order it is presented in the evaluation timestamps labels.

The results for all audio segments to be evaluated should be enclosed in a single .txt file named prediction.txt.

For each audio segment to be evaluated, the audio segment id should take the form of:

{audio name}_{utt id}_{start}_{end}

where the fields are the values given in the timestamp labels released with the MERLIon CCS Evaluation set for Task 1 (Language Identification).

Let’s take the first audio segment in the MERLIon CCS Development set as an example:

In the prediction.txt, the above audio segment should be named as:

TTS_P91182TT_VCST_ECxxx_01_AO_48503281_v001_R004_CRR_MERLIon-CCS_a1_1170_2750

In the prediction.txt, there are two formats allowed. The first format requires each audio segment to have separate English and Mandarin scores per line, where each line has 3 space-delimited fields. The English score (indicated by 0) must always precede the Mandarin score (indicated by 1). For instance:

In the second allowed format for Task 1 (Language Identification), each audio chunk is accompanied by their English and Mandarin scores in the same line, where each line has 3 space-delimited fields.

The prediction.txt which consists of all the prediction scores for all audio segments to be evaluated should then be placed in a zip folder. The zip folder should not contain any spaces in its folder name. For instance, Result For Task 1.zip will be rejected, and result_for_task_1.zip is the accepted format.

As such, the submission zip file must have the following structure:

Note that the prediction.txt must be directly in the results.zip. The following structure will be rejected:

WRONG FORMAT

Results Zip Archive Format for Task 2

For Task 2 (Language Diarization), a separate RTTM file should be generated for each audio recording. The RTTM file for each audio file should be named according to the audio filename. For instance, for the audio recording:

TTS_P12345TT_VCST_ECxxx_01_AO_12345678_v001_R004_CRR_MERLIon-CCS.wav

The corresponding RTTM file should be named:

TTS_P12345TT_VCST_ECxxx_01_AO_12345678_v001_R004_CRR_MERLIon-CCS.txt

In the RTTM file, each line contains three space-delimited fields, start time, end time and language id, indicating the onset and offset of language turns in milliseconds. For example:

All RTTM files to be evaluated must be placed in a zip folder. The zip folder should not contain any spaces in its folder name. For instance, Result For Task 2.zip will be rejected, and result_for_task_2.zip is the accepted format.

The accepted zip folder structure for Task 2 submission is as follows:

Note that the RTTM files must be directly in the results.zip. The following structure will be rejected:

WRONG FORMAT

Please follow our formatting instructions carefully. CodaLab will not be able to score your submission if it is not formatted as such.

When you’re zipping the files, please do not zip a folder containing all the files. All files should be selected and then zipped, I.e., files must reside directly in the zip folder without an additional folder inside the zip folder. See wrong formats above.

If you are using a MacOS, please use a 3rd party application to zip the file, instead of the “Compress” function on your MacOS. Otherwise, an extra folder appears (per the figure below), which will disrupt the file hierarchy of your submission, and CodaLab will be unable to process your submission.

Submitting Results on CodaLab

Results Submission

        ·   # -- ordinal number of submission in system; your first submission will be listed as 1

·       SCORE -- DER for the submission; if the scoring is in progress or failed, this will read "---"

·       METHOD NAME -- the name of the system that produced the submission

·       FILENAME -- name of the zip file you submitted

·       SUBMISSION DATE -- date and time of submission in MM/DD/YYY HH:MM:SS format (all times are UDT)

·       STATUS -- the current status of your submission, which may be one of

·     Submitting -- zip file is being uploaded

·     Running -- upload is successful and scoring script is running

·     Finished -- scoring script finished successfully and results posted to leaderboard

·     Failed -- scoring script failed

·       checkmark -- indicates whether or not submission is on the leaderboard

Figure shows example of your screen after a submission.

·       You can click “Refresh Status” to stay updated on the status of your submission.

·       If everything is formatted properly, Codalab will be able to accept and score your submission.

·       In the case that there could be processing delays in CodaLab, it might take a significant amount of time (up to 1 hour) for your submission to be scored.

·       If your submission has taken longer than that to process, let us know at merlion.challenge@gmail.com, and we will assist you with your submission.

 

If scoring failed for your submission, click the + symbol to the right of its entry in the table. This will display the following, which may be used for debugging purposes:

·       Method name -- the method name you entered into the form

·       Download your submission -- a download link for the zip file submitted

·       View scoring output log -- the scoring program’s output to STDOUT

·       View scoring error -- the scoring program’s output to STDERR

·       Download output from scoring step -- ignore; downloads a zip file containing files used by CodaLab internally

·       If scoring failed, please double check that you have followed all our instructions for formatting your file before submission.

·       Only three submissions are allowed per day per user. Please make sure your submissions are formatted correctly before submitting.

·       While the submission limit do not count towards submissions that are not scored successfully, the organizers will have to delete failed submissions in the backend when the limit is reached. Please reach out if you need help.

Leaderboard

·       After your submission finishes scoring (status “Finished”) it will post to the leaderboard, which is viewable from the Results tab.

·       The leaderboard lists the most recent submission for each system by each team, ranked in ascending order by DER.

·       For each submission on the leaderboard, the following fields are displayed:

·       # -- ranking of system

·       User -- the username for the account that submitted the result

·       Entries -- total number of entries by account that submitted result

·       Date of Last Entry -- date of last entry by user that submitted result in MM/DD/YY format

·       Team Name -- name of team associated with user that submitted result; this is taken from the Team listed on the user’s profile

Task 1 (Language Identification):

·       Equal Error Rate – The primary metric for Task 1 (Language Identification)

·       Balanced Accuracy

Task 2 (Language Diarization):

·       Total Language Diarization Error Rate – The primary metric for Task 2 (Language Diarization)

·       English LDER

·       Mandarin LDER

Leaderboard Example

Interspeech 2023 Paper Submission

Note: Submitting a paper is optional, however, if you are interested in doing so, the instructions are detailed below.

Go to the main Interspeech 2023 website, and click on Paper Submission under “For Authors”.


Scroll down the page, until you get to the link “Click Here to Submit Your Paper”.

Once you click hat, you’ll need to create a Microsoft CMT Account.

After creating an account, you will be asked to fill in your domain conflicts. Please fill in your domain conflicts, as well as the domain of the institution you currently belong to:

After you are done with filling in your domain conflicts, click the link below to create a new submission.  

Select "INTERSPEECH 2023 MAIN TRACK".

Scroll down to the Section labelled “Subject Areas”, and select our challenge. You can find it under Special Sessions and Challenges.  

Scroll to that section and select “14.11 MERLIon CCS Challenge: Multilingual Everyday Recordings - Language Identification On Code-Switched Child-Directed Speech”.  

Fill in the rest of the page, and click submit at the end of the page, like so:  

System Description Submission

Each submitted system to the challenge must be accompanied by a system description which sufficiently details the system in a way for a fellow researcher to understand the approach and resources required to train and run the system. For more information, please refer to Appendix D of our Evaluation Plan.

Please submit your systems description to merlion.challenge@gmail.com, with subject title “Systems Description – [Your Team Name]”.

For example, if your team name is “Merli”, please send your system description in an email with subject title “Systems Description – Merli".

System Description Submission should be done within 24 hours after the Interspeech Paper Submission deadline.


For more information on the result output formatting guidelines and submitting results on CodaLab, please refer to the instruction booklet here.