First draft - Colin White & Frank Hutter
Tabular NAS Benchmarks
Released in 2019
423k architectures, each trained 3 times on CIFAR-10
The first queryable benchmark for NAS. The release of NAS-Bench-101 substantially improved best practices for NAS research, by making it easy to compare algorithms across many trials, and also by fixing the training and evaluation pipelines of architectures.
Follow-up, NAS-Bench-1shot1, allows for benchmarking one-shot algorithms.
Released in 2019
6k unique architectures, each trained 3 times each on CIFAR-10, CIFAR-100, and downsampled ImageNet.
The first queryable benchmark that contained multiple datasets and the full learning curve.
Surrogate NAS Benchmarks
Surr-NAS-Bench-DARTS (NAS-Bench-301)
Released in 2020
Search space is size 10^18.
The first surrogate NAS benchmark, which allows NAS benchmarks on search spaces that are many orders of magnitude larger than tabular NAS benchmarks.
Created by training 60k architectures and using them as training data to fit a surrogate model on the rest of the space.
Released in 2021
Search spaces of size 423k, 10^18, and 10^54.
The first surrogate NAS benchmark to include full learning curves for each architecture (rather than only including the final accuracy).
Created by separately modeling the average learning curve, and the noise of the learning curve, and then sampling random noise for each learning curve.
NAS+HPO Benchmarks
Released in 2019
Joint Architecture and Hyperparameter Search on feedforward neural networks on regression problems
Released in 2022
A collection of surrogate benchmarks for Joint Architecture and Hyperparameter Search
Supports multi-objective, cost-aware, and multi-fidelity optimization with several fidelity dimensions.
Hardware-Aware NAS
Released in 2021
Extends NAS-Bench-201 to include hardware information for several settings, such as edge devices.
Benchmark Suites
Released in 2021
Contains 26 tasks across 7 search spaces.
Unifies the API/framework for nearly all NAS benchmarks into one codebase.
Released in 2021
Contains 10 search spaces.
Far more diverse than prior NAS benchmarks.
Includes data from CV to earth monitoring to astronomy to PDE solving to medical diagnostics.