Conformal Inference
Conformal Inference
Embedded Files
Machine learning and deep learning models are powerful prediction tools with many successful applications in domains such as image recognition and financial stock price forecasting. Most works in this field have aimed to produce the most accurate possible point predictions, but relatively few focused on capturing the uncertainty associated with those predictions. Understanding predictive uncertainty in machine learning can be difficult, but it is an essential problem in many high-stake applications such as disease detection and autonomous driving.
A promising approach to address this challenge is made possible by conformal prediction (CP), which is an increasingly popular statistical framework for quantifying the predictive uncertainty by converting the output of any machine learning black-box models into a prediction set/interval of likely outcomes. The main strength of CP is that it can provide provable finite-sample guarantees under minimal data exchangeability assumptions.
[Training Uncertainty-Aware Classifiers with Conformalized Deep Learning]
Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, Yanfei Zhou
The Conference and Workshop on Neural Information Processing Systems (NeurIPS) 2022; Poster presentation
In this project, we integrate CP into the training process of deep multi-class classification models by designing a novel conformal loss function called the 'conformal loss'. Training with conformal loss helps reduce model overconfidence, and when combined with standard post-hoc calibration, produces more efficient conformal prediction intervals with more reliable coverage.
For example, the figure below showcases two example test images from the CIFAR10 dataset, respectively intact and corrupted by RandomErasing, along with their corresponding predicted probabilities calculated by deep neural models trained using different loss functions. This shows the model trained with our conformal loss is not as overconfident when dealing with the corrupted images as that minimizing the cross entropy.
Page updated
Report abuse