Xiaojia Guo - bayesian ensemble

Bayesian Ensembles of Unlabeled Forecasts

Xiaojia Guo (Robert H. Smith School of Business, UMD), Kenneth C. Lichtendahl (Google LLC), Eric Tassone (Google LLC)

In combining forecasts from several experts or models, the trimmed mean often offers improvements over the simple mean because it removes extreme forecasts that can severely bias the simple mean. When one concludes that an extreme forecast is severely biased, the trimmed mean becomes an unsupervised ensemble. It represents a form of unsupervised learning—a way to draw inferences from the current forecasts alone, without any past forecasts or realizations of the quantity of interest. The trimmed mean is a crude form of unsupervised learning because it splits the forecasts into two clusters, puts equal weight on the middle forecasts, and no weight on the extreme forecasts. In this paper, we introduce a more sophisticated unsupervised ensemble. Our ensemble follows from a Bayesian model of the forecasts and the experts’ biases, but without exact knowledge about which expert is least biased, second-least biased, etc., which is often the case in practice. This model learns only from the order statistics of the experts’ point forecasts, as if the forecasts were otherwise unlabeled. In other words, the model learns anonymously, without knowing the identity of the experts and how they performed in the past. According to the model, the more extreme a forecast is, the more likely it is to be highly biased. The form of our ensemble is a linear combination of the forecasts and consequently can be viewed as a robust mean. The forecasts that are more likely to be highly biased get lower weights in our ensemble. In an empirical study of time series forecasts from the M4 competition, we demonstrate that our Bayesian ensemble can outperform the simple and trimmed means and the best combination model from the competition. Our ensemble can also be used to produce prediction intervals or forecast binary events, which makes it a flexible tool for use in practice.

(Work in progress)

Figure 1. Weight Put on an Exponential Smoothing Model (ets) as Its Forecast Goes Extreme. Suppose there are five models in the ensamble and one of them is the ets model. It is easy to see that unlike the other two trimmed mean methods, the weight assigned by the Bayesian method decreases continuously as the forecast goes extreme.