In this section, we perform an empirical study on repairing method, which also inspire us to propose ArchRepair. After investigating the existing repairing method, we find that existed repairing methods are focusing on repairing DNN model either on neuron-level (repairing on certain neurons) or network-level (usually using augmented data to fix all weights of the network), where the former one might be ineffective while the later one could be time-consuming. We also find the existed repairing methods repair DNN model by adjusting its weights without modifying the architectur, therefore, we perform this empirical study to explore the possibility of repairing DNN model by adjusting its architecture on block-level. The results show that repairing DNN on block-level have its own advantages, which make a balance between effectiveness and efficiency.
First we evaluate the effectiveness of the repairing method from the perspective of the architecture level. We select 3 popular block-based networks (i.e., ResNet-18, ResNet-50, and ResNet-101) as targets. We use the trained DNN model to generate a failure dataset from original training dataset, which only includes misclassified data. We divide the architecture into 4 different levels: Neuron-level, Layer-level, Block-level, and Network-level. For each architecture level, we randomly select 100 failure cases and use them to calculate the gradients. Then we select the repairing target (neurons in Neuron-level, layer in Layer-level, block in Block-level) which has the greatest gradients and use the original training dataset to retrain them. The results show in following tables.
On ResNet-18, we find that although repairing on network-level gets the highest accuracy, it takes 194% of the reparing time comparing to the block-level. Besides, when repairing a larger scale DNN, (i.e., ResNet-101, which usually needs more time on training), repairing on Block-level can save approximately 2700s (about 28% of total repairing time) while achieving a competitively high accuracy after repairing. Therefore, we believe repairing on Block-level can make a balance between efficiency and effectiveness.
Besides, comparing the results from the perspective of architecture, we find that repairing on ResNet-101 can always get a higher accuracy than on ResNet-18 and ResNet-50, and the accuracy on ResNet-50 is also higher than which on ResNet-18. These facts indicate that the architecture of DNN can also affect the model's accuracy on dataset, motivate us to investigate reparing DNN through optimizing the architecture.