Remote, Tel Aviv time
Neural Architecture Search (NAS) has become increasingly important for many computer vision systems to automate the design of neural network architectures. However, due to the exponential size of the search space, many classical NAS algorithms require hundreds of GPU days which is not practical for standard users. Recently, significant progress has been made to improve the efficiency of NAS and make running NAS possible even for regular users on standard GPU machines. This tutorial aims to introduce and summarize recent progress in efficient NAS which enables the application of NAS to a diverse set of tasks.
We will start with a brief introduction to NAS and go over its history.
This part introduces the family of differentiable NAS algorithms, which speeds
up NAS by continuous relaxations and weight sharing. Starting with the seminal
work of DARTS, we will introduce recent improvements in supernet training,
architecture selection, and scalability.
This part introduces the family of predictor-based NAS algorithms, which
iterative fine-tune the surrogate performance predictor and use it to sample
unexplored configurations. We will cover the recent progress including
Graph Neural Network (GNN) based predictors, improved Bayesian sampling,
and efficiency improvements by successive halving.
This part introduces a new class of NAS algorithms proposed recently which
estimate the performance of architectures based on some zero-shot properties,
such as some graph-based features for neural networks, expressiveness
of networks, and the condition number of the Neural Tangent Kernel (NTK).
We will introduce several applications of NAS in real-world computer vision
problems, such as a transformer or CNN-based classification, detection, and
semantic clustering. We also present how NAS inspires the development
of other AutoML tasks such as Automated Optimizer Search.
This part will conclude the current progress of NAS and point out important
future directions for researchers to work on.