Summary: We developed a prediction interval method, called PI3NN, to quantify machine learning (ML) model prediction uncertainty and integrated it with Long Short-Term Memory (LSTM) networks for robust time-series forecasting and data shift characterization.

Accomplishment: LSTM networks have been widely used for sequential data learning and time series prediction. However, they can neither quantify uncertainty nor identify the possible data shift caused by changing environmental or operational conditions, resulting in overconfident prediction and misguidance of decision making. Researchers developed an uncertainty quantification method, called PI3NN, to address these problems. PI3NN calculates Prediction Intervals by training 3 Neural Networks and uses root-finding methods to determine the interval bounds precisely. Additionally, it can accurately identify the out-of-distribution (OOD) data in a nonstationary condition to avoid the overconfident prediction. The integrated PI3NN-LSTM method has been applied to several watersheds across US for streamflow predictions under the changing climate. Results indicate that for the prediction data which have similar features as the training, PI3NN precisely quantifies the prediction uncertainty with the desired confidence level; and for the OOD data where the LSTM network fails to make accurate predictions, PI3NN can produce a reasonably large uncertainty indicating the untrustable result to avoid overconfidence. PI3NN is computationally efficient, reliable in training and generalizable to various network structures and data with no distributional assumptions. It can be broadly applied in ML-based time series prediction for trustworthiness and robustness.

Reference: https://doi.org/10.1002/essoar.10512253.1