An OOD detector is a method that identifies the given data to be ID (following the same distribution as the training set of a given model) or OOD (following a different distribution compared to the training set).
Embedded Files
MSP
MSP
The Maximum Softmax Probability (MSP) is a baseline method to detect wrongly classified and OOD samples. The intuition behind MSP is that a model tends to be rather confident (have a high softmax score) on correctly classified data. In concrete, given the softmax distribution of a model's prediction on a test sample, MSP identifies the data as ID if the softmax score is greater than a threshold and vice versa.
ODIN
ODIN
Similar to MSP, the Out-of-DIstribution detector for Neural networks (ODIN) also takes advantage of the softmax distribution. It points out that by adding slight perturbations to samples, the gap in softmax score between wrongly and correctly classified samples becomes larger. Compared to MSP which directly uses the original data to obtain the softmax score, ODIN first preprocesses the data by adding a perturbation and then scales the softmax output by a temprerature scaling parameter. Following the original implementation, we use different perturbation magnitudes (0, 0.0005, 0.001, 0.0014, 0.002, 0.0024, 0.005, 0.01, 0.05, 0.1, 0.2) and set the scaling parameter to 1000.
Mahalanobis
Mahalanobis
The Mahalanobis detector learns a Gaussian distribution for each class based on the training set. Then the detector calculates a confidence score for a test sample by measuring the Mahalannobis distance between the sample and the closest class-conditional Gaussian distribution. The test data is ID if its confidence score is greater than a threshold and vice versa. Similar to ODIN, this detector also preprocesses the test samples with a perturbation. Following the original implementation, we use different perturbation magnitudes (0, 0.0005, 0.001, 0.0014, 0.002, 0.0024, 0.005, 0.01, 0.05, 0.1, 0.2$).
OE
OE
Different from the above three detectors, the Outlier Exposure (OE) detector trains a neural network to detect OOD data by using available ID and OOD data. Remarkably, the OOD data for training is not necessarily to include the distribution of test OOD data, which makes the OE detector more practical to generalize to unseen distributions.
Detector illustrations
Detector illustrations
Figure 9. MSP detector
Figure 10. ODIN detector
Figure 11. Mahalanobis detector
Figure 12. OE detector
Page updated
Google Sites
Report abuse