Description Figure 1: Example MNIST image, and the minimally transformed images required to change the label of each classifier classifier. The distance is indicated in the title of each subfigure. Figure 2: Example CIFAR-10 image, and the minimally transformed images required to change the label of the classifier. The distance is indicated in the title of each subfigure. For a given image, we measure the robustness of a classifier relatively to the transformation group as the minimal normalized distance between the identity transformation and a transformation that changes the classification label when applied to the image. A global invariance measure is then defined as the expectation over the data distribution. A crucial choice in the above definition is that of the metric d. Our novel key idea is to represent the set of transformed versions of an image as a manifold; the transformation metric is then naturally captured by the geodesic distance on the manifold. For a given image, the invariance measure therefore corresponds to the minimal normalized geodesic distance on the manifold that leads to a point where the classifier's decision is changed.The geodesic distances are computed numerically using the Fast Marching algorithm (R. Kimmel, J.A. Sethian, PNAS 1998), and the algorithm is stopped whenever a transformation that changes the classifier's decision is visited.Download codeMATLAB implementationDownload MANITEST code v1.1 (+ stored models used in BMVC paper). Last updated: 9 Aug 2015.C++ with OpenCV implementationLast updated: 17 Aug 2015.PublicationsManitest: Are classifiers really invariant? Alhussein Fawzi, Pascal Frossard. Proceedings of the British Machine Vision Conference (BMVC), 2015.[pdf] [extended abstact] BibTeX: @inproceedings{fawzi15manitest, author = {A. Fawzi and P. Frossard}, booktitle = {Proceedings of the British Machine Vision Conference (BMVC)}, title = {Manitest: Are classifiers really invariant?}, year = {2015} } QuestionsIf you have any questions or comments regarding this work, feel free to contact Alhussein Fawzi (alhussein.fawzi AT epfl.ch). |