Embedded Files
Abstract
Abstract
Over the past decade, deep learning (DL) has achieved a big performance leap in company with the booming of big data. While it has been successfully applied to some industrial domain-specific tasks (e.g., face recognition, speech recognition), its quality and reliability raise great concern especially in the context of safety- and security-critical scenarios. Adversarial examples (AEs) represent a typical and important type of defects needed to be urgently resolved, on which a DL software makes incorrect decisions. Such defects can occur through either intentional manipulation of adversarial attack or physical-world noise perceived by input sensors, potentially hindering the industry deployment.
The intrinsic uncertainty nature of deep learning decision could be a fundamental reason for its incorrect behavior. Although many testing, adversarial attack and defense techniques have been proposed, it still lacks a systematic study to uncover the relationship between AEs and DL uncertainty. In this paper, we conduct a large-scale study towards bridging this gap. We first investigate the capability of multiple uncertainty metrics on differentiating natural benign examples (BEs) and AEs. Then, we identify and categorize the uncertainty patterns of AEs and BEs, and find that while natural BEs and AEs generated by existing methods do follow common uncertainty patterns, some other uncertainty patterns are largely missed. Based on this, we propose an automated testing technique to generate multiple types of uncommon AEs and BEs. Our further evaluation reveals that the uncommon data generated by our methods is hard to be defended by the state-of-the-art defense techniques with the average defense success rate reduced by 35%. Our results call for attention to generate more diverse data for evaluating and designing quality and reliable assurance solutions for DL software.
Overview of study design and approach
Overview of study design and approach
The above figure shows an overview of our work, comprised of two major components:
- an empirical study about the uncertainty metrics and characteristics of the existing data
Reliability issues and security concerns of DL softwares are continuously hindering their more widespread applications into safety-critical systems. The incorrect behavior and quliaty issues of DNN on some inputs could be caused by its intrinsic uncertainty. Hence, we first perform a series of studies to explore the relation between adversarial defects and DL systems from the lens of uncertainty, including:
- an empirical study towards understanding the capability of different uncertainty metrics on distinguishing AEs and BEs. There are different kinds of metrics that could be used as uncertainty representative. We adopt the AUC-ROC score to evaluate the effectiveness of uncertainty metrics on distinguishing adversarial and benign data.
- a study towards the uncertainty patterns of the natural BEs and AEs generated by existing adversarial attack/testing tools. Based on the evaluation result of different uncertainty metrics, we present a way of categorizing AEs and BEs based on the selected uncertainty representatives- PCS and VRO. We then evaluate natural benign data and adversarial inputs generated through attacking methods and testing tools to study their uncertainty patterns on a benchmark of four models across three datasets.
- the data generation algorithm which generates data with uncommon patterns and further applications
The empirical study results show the common uncertainty patterns that existing AEs/BEs usually fall into and also reveal the usefulness of testing tools on generating AEs with different behaviors. However, there are data samples with uncommon patterns which are still largely missed by existing methods. These data samples have the potential to uncover the unknown behaviors of DL softwares. Hence, we developed a tool, KuK (Know the UnKnown) to generate such uncommon data samples.
- we propose a genetic-algorithm-based approach to generate data with uncommon patterns. The test data generation of umcommon patterns is a complicated optimization problem due to the atypicla behavior of such data. We leverage the genetic algorithm to find the solutions, guided by the uncertianty representaives.
- we evaluate the effectiveness of generated uncommon data samples on bypassing the current state-of-the-art defense techniques. Data generated by existing methods and data generated by KuK are denoted as common data (Comm) and uncommon data (UnCo), respectively. We perform the defense techniques on these two data groups to evaluate the usefulness of generated data on revealing errors of DL systems.
Google Sites
Report abuse