Performance measures

Accuracy, Precision, specificity, sensitivity, and support

There are the following performance measures that we need to introduce:

• Accuracy: How often classifier is correct

• Precision: How many selected item is relevant

• Specificity or selectivity: Proportion of actual negatives correctly identified

• Recall or Sensitivity: How many relevant items are selected

• F1 score: harmonic mean of precision and recall

(2×Precision×Sensitivity)/(Precision+Sensitivity)

• Support: number of occurrences of each label

The confusion matrix seems to be very helpful for understanding the performance of the model on the validation (or test) set. However, there are other measures that can give further understanding of the model performance. Here we list a few.

Accuracy is simple and says how often the classifier is correct. This number can be between 0 to 100 percent. If we have two cases, then any accuracy above 50 percent can be considered since it is better than random selection.

Precision says how many selected items are relevant. In other words, it is the proportion of what we correctly identified as positive over what we in general identified as positive (subjective) i.e., proportionally making fewer errors.

Recall of sensitivity, means, how many relevant items are selected. In other words, the proportion of what we correctly identified as positive over what is, in reality, positive (objective) i.e., proportionally better reflecting the reality.

F1 score is a measure that introduces a balance between precision and sensitivity and is introduced as the harmonic mean of precision and recall.

Support is the number of occurrences of each label.

We can show the precision, specificity, and sensitivity in terms of the components of the confusion matrix. As it has been discussed earlier that the threshold of a probabilistic classifier is important in the introduction of a classifier. There are other parameters that can also have a direct and indirect impact on the classifier. However, the decision-maker can have different concerns. He/she can be more concerned with precision, specificity, or the sensitivity of the classifier. For instance, if an insurance company wants to pay a claim, it needs to be more precise or specific, so it needs a classifier to classify the claim with high precision or specificity. While in another problem considering the credit card business, in order to avoid any fraudulent transaction the banker needs to be sensitive about any fraud report.

There is always a trade-off between these measures and, for example, it is impossible to have a highly precise and highly sensitive classifier. This is based on the decision-maker's decision to decide about any measure. But if the decision-maker wants to consider a balance of the measures, she/he can look at the F1-score. But that is not the only measure. We will discuss more measures later.