Notes: Word Error Rate

The word error rate (WER) captures the performance of an Automatic Speech Recognition (ASR) engine. It is defined as the number of words in error in the transcription (output of an ASR) when compared to a reference text.

For example, if the reference text is

"My name is Sunil Kopparapu"

and the ASR output when I speak

/my name is Sunil Kopparapu/

is

"what name sunil kumar kopparapu".

Then WER is computed as the number of words in error when "what name sunil kumar kopparapu" is compared with "My name is Sunil Kopparapu".

In the above example, there are 5 words in the reference text and there are errors in the form of

1. "my" has been substituted by "what",
2. "is" in the reference text has been deleted and
3. "kumar" has been inserted.

Note that generally there are three types of errors, as seen above. So in all there are 3 words in error and the number of words in the reference text is 5. So

WER =3/5 = 0.6

So the WER is 60%.

Our analysis on call center conversation (typical calls between an agent and a customer) show that for a decently trained speech recognition engine the WER is pretty high.

The histogram shows the distribution of WER on some 48 real call center conversations. It is very clear that the WER for natural language conversations is very poor. Typically the WER is >50%. However, with fine tuning the ASR in terms of acoustic modelling and statistical language modelling, the WER can be improved. For example, see the histogram below. The blue bars correspond to pre training and the brown bars correspond to the WER after training. In this case, the training was only in terms of fine tuning the language model.

Observer that the number of speech conversation samples with greater WER have reduced (All the brown bars have shifted left towards lower WER).