Lecture Notes

I will post the handwritten notes that I make for myself here. I do not proofread them carefully, they are written primarily to help me prepare for class, they are not intended to be the official notes for the class. It is important for you to come to class and take notes, and read the textbook and other references, one of which is available online. Please do not ask me to put up the notes before lectures, I may not always be able to do this.

• Applications

• The Basics

• Some Linear Algebra Facts

• Convex Sets and Functions

• Examples of Convex Functions

• Gradient Descent

• Descent Lemma

• Convergence: Fixed Stepsize

• Convergence: Armijo's Rule

  • Application: Neural Networks and The Backpropagation Algorithm

• Newton's Method

• Projected Gradient Descent

• Lagrange Multipliers: Equality Constraints

• Lagrange Multipliers: Inequality Constraints

• Lagrange Multiplier Examples

  • Application: Wireless Power Allocation

• Penalty and Barrier Function Methods

• Application: Internet Congestion Control

• Duality

  • Application: Support Vector Machines

  • Application: Zero-Sum Matrix Games

• Augmented Lagrangian Method

• Subgradient Descent

• Proximal Gradient Method

• Online Convex Optimization