In this section, we discuss the robustness of models repaired by ArchRepair. In section RQ1 we repair the DNN model on clean dataset and evaluate on corruption datasets, finding that ArchRepair can enhance model's robustness. Therefore, in this section we will answer the question that when repairing the model trained on a specific corrupted dataset, can ArchRepair also improve the overall accuracy without harming the robustness on other corrupted datasets.
In this section, we also compare ArchRepair with other 6 SOTA repairing methods on 4 different DNN model (i.e. ResNet-18, ResNet-50, ResNet-101, and DenseNet-121) and 2 popular dataset (i.e. CIFAR-10 and Tiny-ImageNet) with their corresponding corruption datasets (i.e. CIFAR-10-C and Tiny-ImageNet-C). For each pre-trained DNN model, we also first generate the repairing dataset, however, the failure cases is collected from one corruption dataset instead of from original clean dataset. Then we use this repairing dataset to repair the models and evaluate their performance on original testing set and corrupted datasets. The results are shown in following tables.
In summary, ArchRepair performs better than other repairing methods on 8 out of 12 corrupted datasets (i.e. Gaussian Noise, Shot Noise, Glass Blur, Motion Blur, Snow, Frost, Contrast, and Pixelate), demonstrating that ArchRepair is capble of repairing the model on common corrupted dataset. Meanwhile, we also find that no matter evaluating on which corrupted dataset, the accuracies of model repaired by ArchRepair are all increased, especially on Impulse Noise corruption, which performs better than the original model and models repaired by other repairing methods on the most of other corrupted datasets, indicating that ArchRepair can enhance model's robustness.