Ranking comparisons

Introduction

The following figures and the accompanying text present a brief demonstration of the impact of the methodology used to aggregate the errors from the four distinct elements. Thus, this page addresses the second challenge faced while evaluating regression model performance. The results below serve as an extension of the official evaluation metric of the contest defined at the launch of the contest. As such, the results below do not affect the final ranking.

About the performance metrics

The considered performance metrics combine the error values determined in terms of RMSE, MAE, RMSRE, MARE, RRMSE, or RMAE for the four distinct elements and combine these into a single value. Each performance metric is strictly non-negative and a value lower value is indicative of better performance. Details of the individual performance metrics and the ranking yielded by them are presented below.

Mean ranking

Average ranking across the four elements. Note, that this performance ranking considering RMSE values yielded the official ranking of the contest.

Mean error metric

Average of the error metric values across the four elements.

Mean error metric scaled to unit maximum

The average error metric value after scaling them to unit maximum. This performance metric aims at expressing how much better a team did compared to the worst-performing team.

Mean error metric scaled to unit median

The average error metric value after scaling them to unit median. This performance metric aims at expressing how much better a team did compared to the average team for the given element.

Mean standardised metric

This performance metric expresses the teams' performance in terms of standard deviations from the mean performance. Since the teams' performances were distributed according to an exponential distribution (most teams were close to 0 error), the error metrics have been log-transformed prior to standardizing them to unit zero mean and unit variance. Consequently, the distributions have been translated to zero minimum to avoid negative values.

Relative performance

The average error metric value after scaling them to unit minimum. This performance metric aims at expressing how much worse a team did compared to the best-performing team.