ELE539/COS512: Optimization for Machine Learning

Spring 2023,     Mon. Wed. 09:30 am - 10:50 am, Friend Center 004

• Instructor:   Chi Jin                Office hour:  Mon. 4:00 pm - 5:00 pm

• TA:                Ahmed Khaled      Office hour:  Tue. 4:00 pm - 5:00 pm

• Contents:     Optimization theory---algorithms and complexity analyses

• Grades:        5 problem sets (60%), 1 final exam (40%). 

• No late homework. 

Schedule (weekly basis)

Convex Optimization: 

1. Intro, black-box model

2. Subgradient methods, gradient methods

3. Nesterov's acceleration, momentum

4. Lower bounds                                                 [Homework 1 due]

5. Mirror descent

6. Stochastic algorithms, variance reduction                                                      [Homework 2 due]

Nonconvex Optimization

1. Eigenvector problems, power methods, Lanczos algorithm

2. Finding stationary point                                                                                     [Homework 3 due]

3. Escaping saddle point

4. Nonsmooth (weakly convex) optimization                                                      [Homework 4 due]

Minimax Optimization

1. Convex-concave: gradient descent ascent, extragradient method

2. Gradient descent with max-oracle                                                                   [Homework 5 due]

Recommended Textbook

• Convex Optimization: Algorithms and Complexity, by Sebastian Bubeck

• Lecture Notes: Optimization for Machine Learning, by Elad Hazan

• Convex Optimization, by Stephen Boyd and Lieven Vandenberghe

Related Courses

• Elad Hazan, Optimization for Machine Learning

• Yuxin Chen, Large-Scale Optimization for Data Science