Optimization Boot Camp

Details: Feb 7-Feb 18.

Notes

• 0 (briefly reviewed, mostly a reference) Linear Algebra

• 1: Optimizing quadratic functions

• 2: Convex and nonconvex analysis (typo fixed 2/19)

• 3: GD and SGD

• 4: continuous time view on optimization

• 5: Nesterov accelerated methods and estimate sequences

• 6: Mirror descent and exponentially reweighted algorithm

• 7: Newton's 2nd order method (Updated 2/15)

• 8: Software inspired proofs (Updated 2/19, incorporating some comments from one of the papers' author in the footnote)

Feedback survey: https://forms.gle/W5MJdq7tbDbt9hqU8

Important note: While most of the subjects cover techniques for convergence proofs, this is by no means an extensive coverage of all the interesting optimization research topics! The possibilities are endless!

Videos (Stony Brook internal only)

• Day 1

• Day 2

• Day 3

• Day 4

• Day 5

• Day 6

• Day 7

• Day 8

Further reading

• Recommended textbook for linear algebra: Matrix Analysis for Scientists and Engineers, by Alan Laub

• Recommended textbook for optimization, emphasis on first-order methods: First-Order Methods in Optimization, by Amir Beck

• Select lectures from UBC ML course, in particular lectures 4 and 11

• Lectures from UCLA optimization course

• More optimization lectures from UW

• Advanced textbook: Lectures on Convex Optimization by Yurii Nesterov

• Shameless self-plug: my optimization blog

• Acceleration methods by Alex d'Aspremont, Damien Scieur, and Adrien Taylor (FnT text)