PhD Thesis
Received IEEE Biometrics Council Best Doctoral Dissertation Award 2021 Link
Panoptic Defenses for Secure Computer Vision
Agarwal, Akshay; Singh, Richa (supervisor); Vatsa, Mayank (supervisor)
Abstract: As the deployment and usage of computer vision systems increase, protecting these systems from malicious data has also become a critical task. The primary source of information in any computer vision system is the input data, and authenticity of the data is integral to the reliability of a system. With advancements in electronic equipment, especially communication mediums such as mobile phones and laptops, digital data acquisition has become an easy task. Such huge enablements of the cameras and digital contents have raised severe concerns such as capturing unauthorized biometrics data, video voyeurism, and sexting. Apart from that, in the case of person recognition, it is generally seen that when the testing image is captured using the different sensor/camera, the performance significantly drops. In other vital scenarios, digital images are used as evidence in the court of law and criminal investigation. While the image source might be authentic, the image itself might be a spoof or corrupted in a way to fool the machine learning algorithms. The attacks on computer vision algorithms have become advanced enough to trick the machine learning systems and deceive human visual systems. Therefore, proper authentication of digital images and videos is necessary. While many of these challenges of computer vision systems are dealt with individually, this dissertation provides a ‘panoptic’ view to address the challenges ranging from image source identification to the classification of anomalies, using machine learning algorithms. This dissertation focuses on detecting and mitigating the spectrum of attacks on the data level. The four major contributions are (i) sensor identification to ascertain that the image is captured from an authenticated device, (ii) detecting digital attacks, (iii) detecting physical attacks, and (iv) detecting adversarial attacks. In the case of large human identification projects such as India’s Aadhaar project and Integrated Automatic Fingerprint Identification System (IAFIS) of the FBI, a variety of acquisitions devices are used. While it is important to ensure that the images are captured from authenticated devices only, the images captured from these different devices vary significantly in terms of the quality, texture, and illumination, which makes the matching of these images also a challenging task. As the first contribution, we have proposed a camera source identification algorithm and a novel feature selection algorithm to identify the biometric image sensor used for acquisition. The proposed algorithm yields more than 99% classification accuracy on several databases with images captured using multiple cameras. We have also prepared and released two multi-sensor iris databases to promote research on this problem. The next two problems we have addressed in this dissertation are presentation attacks on face recognition systems, through physical presentation attacks and digital attacks such as morphing. A variety of presentation attack instruments have been used, starting from the simple print and replay attacks, to more sophisticated mediums such as silicone masks, latex masks, or wax faces. The proposed presentation attack detection algorithm utilizes a combination of wavelet decomposition and texture feature extraction with support vector machine classifier to distinguish between real and attacked faces. The proposed algorithm outperforms state-of-the-art algorithms, including classifiers based on hand-crafted image features and deep CNN features under several generalized settings, including multiple spectrum. We have also prepared a multi-spectral (i.e., visible, nearinfrared, and thermal) face presentation attack database. It is one of the largest publicly available databases in the physical presentation attack domain. The second contribution focuses on detecting digital manipulations such as morphing and swapping. Morphing is the technique to blend two or more faces to create one morphed image, which can be used to create a duplicate identity, and two individuals can get authorized access using the same identity. We first prepared a large scale database using multiple images collected from multiple mediums such as mobile applications and internet websites. We propose a novel feature extraction algorithm to detect the digital alterations that can encode the artifacts developed due to morphing or swapping. The proposed feature extraction algorithm first filter the image patches and encodes the irregularities as a difference in those local regions. We have observed that because of the sophisticated digital alteration tools, these differences are minute. Therefore, to highlight the irregularities, we assign the weights to the difference value based on its magnitude. Once the features are extracted, a machine learning classifier is trained for binary classification (i.e., real or altered). The massive success of deep convolutional neural networks has significantly increased their usage in machine learning inspired solutions. However, it has been observed that deep learning algorithms are susceptible to intelligently crafted minute noises, and are popularly known as adversarial examples. The adversarial attacks can be both targeted and untargeted. The impact of these adversarial attacks can be seen in the physical world where the simple misclassification of stop sign’ to ‘increase speed’ can cause harm to pedestrian and the autonomous vehicle. Therefore, the detection of adversarial examples is essential for rightful and confident usage of deep learning-based solutions in the real world. As the final contribution, novel detection algorithms are developed to detect different kinds of adversarial attacks. The proposed solutions are the first in the community which can detect such vast and challenging scenarios, and yield the panoptic defense against adversarial examples being agnostic to the databases, adversarial attack algorithms, and CNN architectures.