Abstract: Bayesian Optimization and SMAC

by Marius Lindauer, Difan Deng, Aditya Mohan and René Sass

Bayesian Optimization (BO) is one of the most efficient black-optimization techniques and in particular well suited for hyperparameter optimization and automatic design of ML pipelines. In this hands-on session, we will focus on a simple, yet important application of BO for deep learning: the tuning of the learning rate of deep neural network. The goal of this hands-on session will be that all attendees can implement their own BO approach. Starting with the main loop of BO, we will discuss and implement several surrogate models, acquisition functions and optimization strategies; maybe even a multi-fidelity approach.