Nonconvex Optimization Tutorial

Nonconvex optimization is quickly becoming the most important workhorse of machine learning and finds numerous applications in knowledge discovery and data mining. Examples include deep learning, reinforcement learning, embedding, and Bayesian inference. Nonconvex optimization has been classically known as computationally intractable until recently it witnesses encouraging empirical and theoretical progress. In this tutorial, we first introduce the recent advance on nonconvex optimization theory, especially the analysis of both nonconvex geometry and existing algorithms. Under a unified theoretical framework, we illustrate how such progress inspires the design of new efficient algorithms that are able to solve knowledge discovery and data mining problems that are orders of magnitude larger than previous ones. This tutorial hopes to introduce a new algorithmic framework to practitioners and facilitate multidisciplinary collaboration.