Objective metric & filtering

We perform an objective evaluation on all submissions with a bag of speech recognition systems. We further rank the submissions based on Character Error Rate (CER) and consider the top 10 submissions for subjective evaluation. The details regarding the ASRs to be used are shown below. We choose to use open-sourced ASR models (with <30% WER in their test sets) added to huggingface so that the participants can utilize it to gauge their performance. More models may be added in the next few days. The objective evaluation scripts is shared through the challenge github repository. Note that we use multiple ASRs on a single utterance, consider the transcription with least CER and obtain the corpus CER from the selected transcripts. The score is obtained for all the evaluation sentences.

Hindi ASR

Telugu ASR

Marathi ASR