BIRDS

What is Conformal Prediction?

Conformal Prediction comes under the topic of Uncertainity Quantification, which is quantification of uncertainity. That doesn’t seem very helpful does it? Let me elaborate, in the most simplistic setting of Deep Learning problem we are trying to find an answer. This answer can be predicting the breed of dog, type of object or maybe the next word too. Given some inputs to the model, we get this answer and that’s usually all that we care about. This type of setting is called Point Prediction where point is the answer we get from the model. Uncertainity Quantification can provide us a way to find how certain or confident the model is about these predictions.

Conformal Prediction also provides a way to generate a prediction set for any model. To put it simply:-

Conformal prediction uses an already trained model to estimate “all” predictions that a model can make for the given input with high confidence.

Suppose your model predicts a breed of a dog, with conformal prediction you can get the possible prediction set for that image. So given an image instead of breed as an answer you’ll get a list of breeds possible for that image. The catch is that Conformal Prediction provides us a guarentee that this prediction set fall under a high probablity range that you set. This guarentee is called Coverage.

How do I do it?

So, we know a bit about conformal prediction but how do we do it, say…in code. Well let’s talk about the easier way first, its as easy as cooking instant noodles. You have the tools(libraries) just follow the instructions and you should be good. So, you essentially have 2 steps:-

• Training Step: Training a model on the point prediction task on a dataset.

• Calibration Step: Estimating the Confidence interval/Prediction set of the prediction from the model over a held out IID dataset for this task.

During the calibration step, we do the following:-

• Decide a scoring function and get the score of the correct class. This score could be absolute difference or anything to be fair. So you feed input to the net and let the softmax vector from that vector find the softmax value of the correct label and find its score. Note this softmax value of correct class that the model predicted may or may not be highest.

• Repeat the above for all the points in the calibration set and get the “score list”. Take ~10% quantile of these scores. With this you get a score(let’s call it q) such that 90% points/scores in score list have a true class score more than q.

• Now during prediction, when you get softmax score for the outputs all the score that are more than q form our prediction set.