Course Overview

Optimization is an essential part of modern statistics and machine learning. This course covers topics to develop scalable optimization tools for modern statistics and data analysis. It is designed to help students develop a practical understanding of how and why the existing methods work so that students can learn the core ideas of these methods and use modern statistical methods effectively. This course emphasizes theory and algorithms for smooth and non-smooth convex optimization, stochastic optimization, convex finite sum optimization, and their applications in statistics, including Lasso, group Lasso, total variation methods, and large-scale machine learning methods. This course doesn't focus on programming skills training but rather on the computation complexity theories and ideas behind algorithms. This course will also provide opportunities for exploring frontier research in optimization and machine learning.

Textbook

• Optimization for Data Analysis, by Wright & Recht (Recommended) (Book Website)

• First-Order Methods in Optimization, by Beck (Optional) (Book Website)

• Lectures on Convex Optimization, by Nesterov (Optional) (Book Website)

• High-Dimensional Data Analysis with Low-Dimensional Models: Principles, Computation, and Applications, by Wright & Ma (Optional) (Book Website)

• First-order and Stochastic Optimization Methods for Machine Learning, by Lan (Optional) (Book Draft from Author)

• Convex Optimization, by Boyd & Vandenberghe (Optional) (Online Book and Book Website)

• Numerical Optimization, by Nocedal & Wright (Optional) (Book Website)

Topic Outline

1. Statistical and Machine Learning Models

2. Convex Optimization Theory, including Convex Sets, Convex Functions, Subgradient, Lagrange Duality, KKT Conditions, and Conjugate Function.

3. Deterministic Smooth Convex Optimization, including Gradient Descent, Projected Gradient Descent, Conditional Gradient Descent, Accelerated Gradient Descent, Conjugate Gradient Method, Coordinate Descent, and ADMM.

4. Deterministic Non-smooth Convex Optimization, including Subgradient Descent, Mirror Descent, Proximal Descent Method, Smoothing, Dual Proximal Gradient Method, Primal-Dual Subgradient Method, and Mirror Prox Algorithm.

5. Stochastic Optimization, including Sample Average Approximation, Stochastic Approximation/Stochastic Gradient Descent, Stochastic Mirror Descent, Stochastic Variance Reduced Gradient, AdaGrad, RMSprop, and Adam.

Grading

• Bi-weekly homework assignments (60%): The lowest score will be dropped.

• Final project (40%)