NeuroEvoBench: Benchmarking Evolutionary Optimizers for Deep Learning Applications
π Paper | π Benchmark Code | π Analysis/Data Access
Authors: Robert T. Lange, Yujin Tang, Yingtao Tian
Published at NeurIPS 2023 Track on Datasets and Benchmarks
Introducing NEB - A Benchmark Tailored to Neuroevolution
Written in JAX for accelerated and distributed population rollouts
Leveraging evosax & EvoJAX for access to more than 30 different optimizers and accelerated problem wrappers
Easy to adapt to your own method and application - give it a try (colab)!
Open-source release & data available
For easy comparison without
any reproduction issues
NEB = 4 problem classes & 11 total sub-tasks
BBO: Black-box optim. benchmark (BBOB) and HPO-B
Control: Brax robotic and MinAtar visual control tasks
Vision: F-MNIST/CIFAR-10 classification & MNIST VAE
Sequence: Addition regression and Sequential MNISTΒ
Core Benchmark API
NeuroEvoBench comes with a simple modular task design and API.
At its core are three ingredients:
Policy: Defines the network/BBO substrate to optimize its parameters
Task: Defines the task (e.g. rollout of robot policy / loss evaluation of net)
Evaluator: Ties policy, task evaluation and logging together for EO loop
The evolutionary optimizer method only has to follow the ask-evaluate-tell API common to many black-box optimization toolboxes such as evosax.
Check out the example colab to get started evaluating your method.
Tuned Learning Curves & Sweeps for 10 Baseline EO
Performance of tuned EO on 9 neuroevolution tasks
50 random search tuning trials for 10 EO methods
π We provide all learning curves and tuning experiment results via a Google Cloud Storage bucket. Please check out neuroevobench-analysis for more instructions on how to download and visualize the data.
It is as easy as executing this in the command line: gsutil -m -q cp -r gs://neuroevobench/ .
It is as easy as executing this in the command line: gsutil -m -q cp -r gs://neuroevobench/ .