CLOSE: Curriculum Learning On the Sharing Extent Towards Better One-shot NAS

Problem Overview and Motivation

To alleviate the severe problem of search inefficiency in traditional NAS algorithms, one-shot NAS proposes to share operation parameters among candidate architectures in a “supernet” and train this supernet to evaluate all sampled candidate architectures, which reduces the overall search cost from thousands of GPU days to only a few GPU hours. Despite its efficiency, previous studies reveal that one-shot NAS suffers from the poor ranking correlation between one-shot estimations and stand-alone estimations, which leads to unfair comparisons between the candidate architectures. The excessive sharing of parameters, i.e., the large sharing extent, has been widely regarded as the most important factor causing the unsatisfying performance estimation. But our experiments give a more concrete demonstration on how the sharing extent affects the ranking quality. On one hand, using a smaller sharing extent (more parameters, larger supernet capacity) can alleviate the undesired coadaptation between architectures, and has the potential to achieve higher saturating performances. On the other hand, training the supernet with higher sharing extent than the vanilla one (fewer parameters, smaller supernet capacity) greatly accelerates the training process of parameters, and help the supernet obtain a good ranking quality faster. Based on the above results and analysis, a natural idea to achieve a winwin scenario of supernet training efficiency and high ranking quality is to adapt the sharing extent during the supernet training process.

Method

CLOSENet: A Supernet with An Adjustable Sharing Extent

To enable the adaption of the sharing extent during the training process, we design a novel supernet, CLOSENet, whose sharing extent can be easily adjusted. The key idea behind CLOSENet is to decouple the parameters from operations to enable flexible sharing scheme and adjustable sharing extent. Specifically, we design the GLobal Operation Weight (GLOW) Block to store the parameters, and design a GATE module for assigning the proper GLOW block to each operation.

CLOSE: Curriculum Learning On Sharing Extent

We borrow the idea of curriculum learning to design a novel supernet training strategy CLOSE. Specifically, we initialize the CLOSENet with only one GLOW block at the beginning. This large sharing extent helps us to train the supernet much faster. Then, we gradually add GLOW blocks at preset epochs to reduce the sharing extent. In this way, CLOSE not only accelerates the supernet training, but also improves the saturating ranking quality of the supernet. We also propose two tenchiques, WIT and SRT, to further improve the ranking quality.