Risk and uncertainty are central to forecasting and prediction; it is generally considered good practice to indicate the degree of uncertainty attaching to forecasts. In any case, the data must be up to date in order for the forecast to be as accurate as possible ## Forecasting accuracy[edit]The forecast error (also known as a residual) is the difference between the actual value and the forecast value for the corresponding period. where E is the forecast error at period t, Y is the actual value at period t, and F is the forecast for period t. A good forecasting method will yield residuals that are Measures of aggregate error:
Business forecasters and practitioners sometimes use different terminology in the industry. They refer to the PMAD as the MAPE, although they compute this as a volume weighted MAPE. When comparing the accuracy of different forecasting methods on a specific data set, the measures of aggregate error are compared with each other and the method that yields the lowest error is preferred.
It is important to evaluate forecast accuracy using genuine forecasts. That is, it is invalid to look at how well a model fits the historical data; the accuracy of forecasts can only be determined by considering how well a model performs on new data that were not used when fitting the model. When choosing models, it is common to use a portion of the available data for fitting, and use the rest of the data for testing the model, as was done in the above examples.
A more sophisticated version of training/test set. for cross sectional data, cross-validation works as follows: - Select observation
*i*for the test set, and use the remaining observations in the training set. Compute the error on the test observation. - Repeat the above step for
*i = 1,2,..., N*where*N*is the total number of observations. - Compute the forecast accuracy measures based on the errors obtained.
This is a much more efficient use of the available data, as you only omit one observation at each step for time series data, the training set can only include observations prior to the test set. therefore no future observations can be used in constructing the forecast. Suppose - Select the observation
*k + i*for test set, and use the observations at times*1, 2, ..., k+i-1*to estimate the forecasting model. Compute the error on the forecast for k+i. - Repeat the above step for
*i = 1,2,...,T-k*where*T*is the total number of observations. - Compute the forecast accuracy over all errors
This procedure is sometimes known as a "rolling forecasting origin" because the "origin" (
The two most popular measures of accuracy that incorporate the forecast error are the Mean Absolute Error (MAE) and the Root Mean Squared Error (RMSE). Thus these measures are considered to be scale-dependent, that is, they are on the same scale as the original data. Consequently, these cannot be used to compare models of differing scales. Percentage errors are simply forecast errors converted into percentages and are given by . A common accuracy measure that utilizes this is the Mean Absolute Percentage Error (MAPE). This allows for comparison between data on different scales. However, percentage errors are not quite meaningful when is close to or equal to zero, which results in extreme values or simply being undefined. ## Mean absolute percentage errorThe where The difference between Although the concept of MAPE sounds very simple and convincing, it has major drawbacks in practical application - It cannot be used if there are zero values (which sometimes happens for example in demand data) because there would be a division by zero.
- For forecasts which are too low the percentage error cannot exceed 100%, but for forecasts which are too high there is no upper limit to the percentage error.
- When MAPE is used to compare the accuracy of prediction methods it is biased in that it will systematically select a method whose forecasts are too low. This little-known but serious issue can be overcome by using an accuracy measure based on the ratio of the predicted to actual value (called the Accuracy Ratio), this approach leads to superior statistical properties and leads to predictions which can be interpreted in terms of the geometric mean.
n statistics, the mean absolute scaled error (MASE) is a measure of the accuracy of forecasts . It was proposed in 2005 by statistician Rob J. Hyndman and Professor of Decision Sciences Anne B. Koehler, who described it as a "generally applicable measurement of forecast accuracy without the problems seen in the other measurements."^{[1]} The mean absolute scaled error has favorable properties when compared to other methods for calculating forecast errors, such as root-mean-square-deviation, and is therefore recommended for determining comparative accuracy of forecasts.## Rationale[edit]The mean absolute scaled error has the following desirable properties: **Scale invariance**: The mean absolute scaled error is independent of the scale of the data, so can be used to compare forecasts across data sets with different scales.**Predictable behavior as :**Percentage forecast accuracy measures such as the Mean absolute percentage error (MAPE) rely on division of , skewing the distribution of the MAPE for values of near or equal to 0. This is especially problematic for data sets whose scales do not have a meaningful 0, such as temperature in Celsius or Fahrenheit, and for intermittent demand data sets, where occurs frequently.**Symmetry:**The mean absolute scaled error penalizes positive and negative forecast errors equally, and penalizes errors in large forecasts and small forecasts equally. In contrast, the MAPE and median absolute percentage error (MdAPE) fail both of these criteria, while the "symmetric" sMAPE and sMdAPE^{[4]}fail the second criterion.**Interpretability:**The mean absolute scaled error can be easily interpreted, as values greater than one indicate that in-sample one-step forecasts from the naïve method perform better than the forecast values under consideration.**Asymptotic normality of the MASE:**The Diebold-Mariano test for one-step forecasts is used to test the statistical significance of the difference between two sets of forecasts. To perform hypothesis testing with the Diebold-Mariano test statistic, it is desirable for , where is the value of the test statistic. The DM statistic for the MASE has been empirically shown to approximate this distribution, while the mean relative absolute error (MRAE), MAPE and sMAPE do not.^{[2]}
## Non seasonal time series[edit]For a non-seasonal time series, ^{[3]}
where the numerator ## Seasonal time series[edit]For a seasonal time series, the mean absolute scaled error is estimated in a manner similar to the method for non-seasonal time series:
The main difference with the method for non-seasonal time series, is that the denominator is the mean absolute error of the one-step "seasonal naive forecast method" on the training set, This scale-free error metric "can be used to compare forecast methods on a single series and also to compare forecast accuracy between series. This metric is well suited to intermittent-demand series When comparing forecasting methods, the method with the lowest MASE is the preferred method. |