Summary: We developed a distribution-free, computationally efficient, and practically reliable prediction interval method to quantify machine learning (ML) model prediction uncertainty.

Accomplishment: We proposed a prediction interval method (PI3NN) to quantify ML model prediction uncertainty and theoretically proven that it precisely captures the uncertainty for a given confidence level and completely avoids the crossing issues suffered by the state-of-the-art methods for a regression problem y=f(x)+ε. PI3NN is particularly suitable for scientific ML (SciML). The produced uncertainty bound can assess the model prediction’s credibility and trustworthiness; it can identify data/domain shift and understand when the learned model could fail and why it fails; and it also can guide data collection to automate the experimental design and inform decision making. We developed PI3NN to accurately quantify model prediction uncertainty by training three neural networks (NNs), one NN to approximate the prediction value, the other two NNs to estimate the lower and upper bound of the prediction interval. The method directly communicates uncertainty using intervals which provides understandable information for decision making. PI3NN does not introduce any unusual hyperparameters resulting in stable performance and it requires no distributional assumption on data which makes it widely appliable for credible predictions in natural sciences and safe operation and control in complex engineered systems.

Reference: https://arxiv.org/abs/2108.02327