Manitest: Are classifiers really invariant?

Description 



Figure 1: Example MNIST image, and the minimally transformed images required to change the label of each classifier classifier. The distance is indicated in the title of each subfigure.



Figure 2: Example CIFAR-10 image, and the minimally transformed images required to change the label of the classifier. The distance is indicated in the title of each subfigure.


Invariance to geometric transformations is a highly desirable property of classifiers in image recognition tasks. Nevertheless, it is unclear to which extent state-of-the-art classifiers are invariant to transformations such as rotations and translations. This is mainly due to the lack of general methods that properly measure such an invariance. We propose Manitest, a rigorous and systematic approach for quantifying the invariance to geometric transformations of an arbitrary classifier. 

For a given image, we measure the robustness of a classifier relatively to the transformation group as the minimal normalized distance between the identity transformation and a transformation that changes the classification label when applied to the image.


 A global invariance measure is then defined as the expectation over the data distribution.


A crucial choice in the above definition is that of the metric d. Our novel key idea is to represent the set of transformed versions of an image as a manifold; the transformation metric is then naturally captured by the geodesic distance on the manifold. For a given image, the invariance measure therefore corresponds to the minimal normalized geodesic distance on the manifold that leads to a point where the classifier's decision is changed.

The geodesic distances are computed numerically using the Fast Marching algorithm (R. Kimmel, J.A. Sethian, PNAS 1998), and the algorithm is stopped whenever a transformation that changes the classifier's decision is visited.

Download code

MATLAB implementation

Download MANITEST code v1.1 (+ stored models used in BMVC paper). 
Last updated: 9 Aug 2015.

C++ with OpenCV implementation

Last updated: 17 Aug 2015.


Publications


Manitest: Are classifiers really invariant?
Alhussein Fawzi, Pascal Frossard. Proceedings of the British Machine Vision Conference (BMVC), 2015.

BibTeX:

@inproceedings{fawzi15manitest,
 author = {A. Fawzi and P. Frossard},
 booktitle = {Proceedings of the British Machine Vision
Conference (BMVC)},
 title = {Manitest: Are classifiers really invariant?},
 year = {2015}
}

Questions

If you have any questions or comments regarding this work, feel free to contact Alhussein Fawzi (alhussein.fawzi AT epfl.ch).