Lectures 2022
Inspired by recent successes towards automating highly complex jobs like automatic programming and scientific experimentation, I want to automate the task of the data scientist when developing intelligent systems. In this talk, I shall introduce some of the involved challenges and some possible approaches and tools for automating data science. More specifically, I shall discuss how automated data wrangling approaches can be used for pre-processing and how both predictive and descriptive models can in principle be combined to automatically complete spreadsheets and relational databases. I will argue that autocompleting spreadsheets is a simple yet highly challenging setting for the automation of data science. Special attention will be given towards the induction of constraints in spreadsheets and in an operations
Recommender Systems are everywhere -- on e-commerce websites, in movie streaming apps, in hospitals, on smartwatches ... just everywhere. The AutoML community, however, has paid little attention to recommender systems even though the goal of AutoML can be seen as a recommendation problem and hence possibly been solved by recommendation techniques. Eventually, AutoML attempts to find the optimal algorithm for a given input, i.e. typically a given dataset. This is similar to recommendations: identify the optimal product for a given user. In the lecture, you will learn about recommendation algorithms and how they could be used for AutoML. In addition, you will learn why recommender systems are a very interesting field of application for AutoML. Recommender-systems algorithms are highly diverse, i.e. very different in their nature, and recommendation algorithms perform highly differently in different scenarios. As such, identifying the optimal algorithm for a given scenario is very difficult and a challenging, i.e. interesting, application for AutoML.
Dynamic Selection and Configuration of Optimization Algorithms: From Hyperparameter Control to AutoML
by Carola Doerr
by Carola Doerr
Many state-of-the-art optimization algorithms use dynamic hyperparameter settings to adjust their search behavior during the optimization process. This allows, for example, to narrow the search for high-quality solutions to the regions that seem most promising at the current state of the optimization, and to converge from exploration to exploitation. Mechanisms to efficiently adjust algorithm behavior have been studied for several decades. They are, however, often restricted to changing the configuration of a single algorithm. In this lecture, we will discuss how the current boost in AutoML techniques allows us to aim higher. By combining hyperparameter control techniques with automated algorithm selection and configuration, we can use data from previous optimization studies to learn how to efficiently chose between different algorithms and their configurations. Without pretending the problem to be solved, we will discuss a (long) list of open issues that need to be addressed to leverage the full potential of this exciting approach. Since powerful optimization algorithms are an important bottleneck towards more efficient use of scarce resources, we hope to inspire participants to join our ambitious journey towards fully dynamic algorithm selection and configuration pipelines.
Training systems with reinforcement learning that perform well in the real-world end is difficult for a number of reasons. One of the main reasons is that the engineer is faced with a multitude of design choices aiming to represent the real world problem into the Partially Observable Markov Decision (POMDP) abstraction, which is insufficient to capture all aspects of the problem. As a result, the engineer acts through trial and error, optimizing the RL system design until satisfactory performance is reached. This is a tiring, time-consuming process. Learning to learn and Auto RL automate parts of this process, allowing users to focus at a higher level design questions. In this lecture we will go over the current established techniques, such as environment, algorithm, representation, and reward learning, and discuss available tools, how and why they work, and when they fail. Finally, since this is the emerging field, we will conclude with the future outlook and open problems facing the field.
Slides
Bayesian optimization is an established framework and popular tool for hyperparameter optimization of machine learning algorithms. While known for its sample efficiency, vanilla Bayesian optimization can not use readily available prior beliefs the practitioners have on the potential location of the optimum. Thus, it disregards a valuable source of information, reducing its appeal to ML practitioners. To address this issue, recent work has focused on developing new methods that incorporate prior user beliefs about the location of the optimum. This results in a conceptually simple framework available to AutoML practitioners in popular Bayesian optimization libraries. This lecture briefly introduces the topic and reviews the state-of-the-art methods. We will show empirical results on optimization of deep neural network hyperparameters and briefly discuss recent theoretical results on regret bounds.
In the past decade, advances in deep learning have resulted in breakthroughs in a variety of areas from computer vision to natural language understanding to speech recognition. While many factors played into the rise of deep learning, the design of high-performing neural architectures has been crucial to its success. Neural architecture search (NAS), the process of automating the design of architectures for a given task, has seen a rapid development with over 1200 papers released in the last two years, resulting in human-designed architectures being outpaced by automatically-designed architectures. In this lecture, we give an easy-to-follow overview of NAS. We survey search spaces, black-box optimization techniques, weight sharing techniques, speedup techniques, extensions, and applications. For each topic, we will describe the fundamentals in addition to cutting-edge trends. We conclude by describing best practices for NAS research, resources, and promising future directions.