ArchRepair: Block-Level Architecture-Oriented Repairing for Deep Neural Networks

This website is the supplementary materials of the paper "ArchRepair: Block-level Architecture-Oriented Repairing for Deep Neural Networks", which takes the presented results further by including experiments not shown in the paper due to the page limitation.

The website is organized as follows:

  • Empirical Study: In this section, we introduce the motivation of our work on block-level reparng through an experiment on the effectiveness and efficiency of repairing a DNN model on different architecture levels. The result shows that block-level repairing can improve the DNN's accuracy with less computation cost.

  • RQ1 Effectiveness: In this section, we conduct comprehensive experiments on both CIFAR-10 and Tiny-ImageNet datasets to evaluate the effectiveness of ArchRepair on both accuracy and robustness metrics, comparing with 6 other different SOTA repairing methods. The results demonstrate that ArchRepair is the most powerful repairing method to the best of our knowledge. In addition, the performance on the corrupted data indicates ArchRepair can also enhance the robustness of DNN models.

  • RQ2 Robustness: In this section, we use ArchRepair to repair DNN models on corrupted datasets. The experimental results comparing with 6 different SOTA methods shows that ArchRepair can improve the performance of a DNN model trained on specific corrupted dataset without harming the robustness on other kinds of corrupted datasets.

  • RQ3 Location: In this section, we evaluate the effectiveness of Vulnerable Block Localization in ArchRepair by comparing the repairing performance on the block localized with and without using Vulnerable Block Localization. The results show repairing the block localized by Vulnerable Block Localization is the most effective way.

  • RQ4 Ablation: In this section, we conduct ablation study of ArchRepair through repairing on different architecture levels. The results indicate that ArchRepair (i.e., block-level reparing) can take all advantages of its components and fully use its repairing capability to improve the accuracy and robustness of the DNN model.

  • Concrete Example: In this section, we take a repaired ResNet-18 as an example to introduce the detailed workflow of ArchRepair.

  • Summary: In this section, we summarize the insights and takeaways of this work.

  • Replication package: We provide the replication package of our work ArchRepair.

Abstract

Over the past few years, deep neural networks (DNNs) achieved tremendous success and have been continuously applied in many application domains. However, during the practical deployment in the industrial tasks, DNNs are found to be erroneous-prone during to various reasons such as overfitting, lacking robustness to real-world corruptions during practical usage. To address these challenges, many recent attempts have been made to repair DNNs for version updates under practical operational contexts by updating weights (i.e., network parameters) through retraining, fine-tuning, or direct weight fixing at a neural level. Nevertheless, existing solutions often neglect the effects of neural network architecture and weight relationships across neurons and layers. In this work, as the first attempt, we initiate to repair DNNs by jointly optimizing the architecture and weights at a higher (i.e., block) level.

We first perform studies to investigate the limitation of whole network-level and layer-level repairing, which motivates us to explore a novel repairing direction for DNN repairing at the block level. To this end, we need to further consider techniques to address two key technical challenges, i.e., block localizing, where we should localize the targeted block that we need to fix; and how to perform joint architecture and weight repairing. Specifically, we first propose adversarial-aware specturm analysis for vulnerable block localization that considers the neuron status and weights' gradients in blocks during the forward and backward processes, which enables more accurate candidate block localization for repairing even under a few examples. Then, we further propose the architecture-oriented search-based repairing that relaxes the targeted block to a continuous repairing search space at higher deep feature levels. By jointly optimizing the architecture and weights in that space, we can identify a much better block architecture. We implement our proposed repairing techniques as a tool, named ArchRepair, and conduct extensive experiments to validate the proposed repairing method. The results show that our method can not only repair and enhance accuracy but also robustness issues, outperforming the state-of-the-art DNN repairing techniques.

Framework of ArchRepair

The figure above is the overall workflow of ArchRepair. To repair a deployed DNN model, we first apply the Vulnerable Block Localization to identify the vulnerable block, following by Architecture-oriented Search-based Repairing. Vulnerable Block Localization first uses repairing dataset to measure neuron suspiciousness on original DNN model, and captures failure cases (the data that model misclassified in the repairing dataset) to calculate the adversarial information on each layer. Then we integrate them on block-level to get the updated suspiciousness of each block. We sort blocks by updated suspiciousness scores in the descending order, and select the top-one (most suspicious) as the repair target. In Architecture-oriented Search-based repairing, we first transform the original block into Directed Acyclic Graph (DAG), then use an adjacency matrix A to control the architecture of the DAG. Therefore, we can relax the discrete block architecture into a matrix in the continuous space and transform the search problem into a optimize problem. We use the optimized architecture matrix to guide discretizing the block architecture and use repairing dataset to train this new block. Finally, we replace the targeted block with the optimized and well-trained one in the deplyoed DNN.