CSE 106 - Optimization and Learning

Syllabus

• 1. Review: Chapters 2 and 3 of Chong and Zak

  2. Differentiation

     • Chapters 5 of Chong and Zak

     • Matrix Differentiation

     • Notes on differentiation (by Marialena Sfyraki)

  3. Optimization

     • Basics of optimization (Chapter 6 of Chong and Zak)

     • Gradient method (Chapter 7 of Chong and Zak)

       • Gradient descent methods (Notes from Princeton University)

     • Newton method (Chapter 8 of Chong and Zak)

     • Solving linear equations (Chapter 12 of Chong and Zak)

  4. Machine Learning

     • Deep learning

       • Forward and Back Propagation (Chapter 13 of Chong and Zak)

       • Convolutional Neural Networks

       • Sequence Models

       • Graph Neural Networks

       • Transformers

Practice Problem Sets

Textbooks and Lecture Notes

Primary

• 1. Mathematical Methods in Data Science, Sebastian Roch

  2. An Introduction to Optimization with applications to Machine Learning, Edwin Chong. Wu-Sheng Lu, and Stanislaw Zak, Wiley, 5th edition.; free online (older) version, Amazon

  3. Linear Algebra and Optimization for Machine Learning, Charu Aggarawl, 2024

       Secondary

• 4. The mathematics for machine learning, Diesenroth, Faisal, and Ong

  5. Understanding Machine Learning: From Theory to Algorithms; Shai Shalev-Shwartz and Shai Ben-David; free online version;  Amazon

  6. Deep Learning, Ian Goodfellow, Yoshua Bengio and Aaron Courville,  Amazon

  7. Calculus and optimization

     1. Calculus, Linear Algebra, and Differential Forms,  Hubbard and Hubbard; online version (perhaps an older edition)   Amazon, Cornell website

     2. Advanced Calculus, Loomis and Sternberg

  8. Vector Calculus, Marsden and TrombaAn Introduction to Optimization, Edwin Chong and Stanislaw Zak, Wiley, 4th edition.

  9. The Science of Deep Learning by Iddo Drori,  Parts of the book is available at the author's website

  10. Patterns, Predictions, and Actions (mlstory.org) by Mortiz Hardt and Benjamin Recht; Amazon

  11. Geometry of Deep Learning: A Signal Processing Perspective by Jong Chul Ye; Amazon

  12. Convex Optimization - Algorithms and Complexity by Sebastien Bubeck; PDF from author's web page

  13. Dive into Deep Learning — Dive into Deep Learning 1.0.3 documentation (d2l.ai) by Aston Zhang and Zachary Lipton

  14. Neural Networks and Deep Learning: A Textbook by Charu Aggarwal; Amazon

  15. Neural networks and deep learning by Michael Nielsen

  16. Linear Algebra and Learning from Data (mit.edu) by Strang

  17. Neural Network Learning (cambridge.org)  by Anthony and Bartlett

  18. Optimization and Randomization

Supplementary Reading

1. NatureDeepReview.pdf (toronto.edu) 

2. Andrej Karpathy blog