Discovering Robust Convolutional Architecture at Targeted Capacity: A Multi-Shot Approach

Abstract

Convolutional neural networks (CNNs) are vulnerable to adversarial examples, and studies show that increasing the model capacity of an architecture topology (e.g., width expansion) can bring consistent robustness improvements. This reveals a clear robustness-efficiency trade-off that should be considered in architecture design. In this paper, considering scenarios with capacity budget, we aim to discover adversarially robust architecture at targeted capacities. Recent studies employed one-shot neural architecture search (NAS) to discover robust architectures. However, since the capacities of different topologies cannot be aligned in the search process, one-shot NAS methods favor topologies with larger capacities in the supernet. And the discovered topology might be suboptimal when augmented to the targeted capacity. We propose a novel multi-shot NAS method to address this issue and explicitly search for robust architectures at targeted capacities. At the targeted FLOPs of 2000M, the discovered MSRobNet-2000 outperforms the recent NAS-discovered architecture RobNet-large under various criteria by a large margin of 4%-7%. And at the targeted FLOPs of 1560M, MSRobNet-1560 surpasses another NAS-discovered architecture RobNet-free by 2.3% and 1.3% in the clean and PGD7 accuracies, respectively.

Problem Overview

Increasing the capacity of topology by width expansion brings consistent robustness improvements. Nevertheless, in actual deployment, there usually exists a capacity budget requirement on the architecture. To discover superior architectures when augmented to the capacity budget by width expansion, we employ neural architecture search (NAS) to search for adversarially robust architectures at targeted capacities. Some studies apply parameter-sharing NAS for robustness. But the model capacity of different topologies cannot be aligned in the supernet, and one-shot NAS favors larger topologies which can be suboptimal at the targeted capacities. As illustrated in the right, Topology 2 is discovered by one-shot NAS since it has a high one-shot reward. However, Topology 1 with a lower reward and capacity in the supernet might outperform Topology 2 when they are aligned to the same capacity.

A motivating illustration. The FLOPs range of topologies in supernets is large. When evaluated in the supernet, topology 2 (green) is better than topology 1 (orange). However, when aligned to the same capacity, topology 1 is better than topology 2.

Method

We propose a novel multi-shot NAS method to search for adversarially robust architectures at targeted capacities, while taking full advantage of the parameter sharing technique. The core of the multi-shot NAS method is to inter- or extra-polate the performances evaluated by multiple “one-shot” supernets of different sizes to estimate the “multi-shot” reward at the targeted capacity. Concretely, we train K supernets and evaluate the architecture in them to get K one-shot rewards. Then, these one-shot rewards together with the corresponding one-shot capacity are used to fit the extrapolation function family. Finally, the fitted function is used to estimate the reward at the targeted capacity.

The overall workflow. Step 1: Adversarially train K supernets with PGD-7 attack. Step 2: Select extrapolation function family based on the average leave-one-out Spearman ranking correlation. Step 3: Conduct architecture search for targeted capacity using the multi-shot evaluation strategy.

Results

Under White-box Attack

We conduct neural architecture search with the proposed multi-shot evaluation strategy. Comparison with baseline architectures under various adversarial attacks on CIFAR-10 is shown below. Our discovered MSRobNets outperform baseline models with the same resource budgets.

Comparison with baseline architectures under various adversarial attacks on CIFAR-10. MSRobNet-CT indicates the search is targeting at FLOPs CT . The “-P” suffix indicates stage-wise architectures discovered by predictor-based search.

Under Black-box Attack

Black-box PGD-100 attack accuracy on CIFAR-10. Adversarial examples are crafted on an independently trained substitute model

and then used to attack the target model. “MSRobNet” is abbreviated as “MSRN”.

Black-box PGD100 attack accuracy on CIFAR-10. Adversarial examples are crafted on an independently trained substitute model and then used to attack the target model. “MSRobNet” is abbreviated as “MSRN”.

BibTex

@article{ning2020discovering,

title={Discovering Robust Convolutional Architecture at Targeted Capacity: A Multi-Shot Approach},

author={Ning, Xuefei and Zhao, Junbo and Li, Wenshuo and Zhao, Tianchen and Zheng, Yin and Yang, Huazhong and Wang, Yu},

journal={arXiv preprint arXiv:2012.11835},

year={2020}

}

Page updated

Google Sites

Report abuse