Resources for Optimization and Learning

Topics

• 1. Conjugate gradient algorithm

     • An Introduction to the Conjugate Gradient Method Without the Agonizing Pain, Jonathan Shewchuk, November 1994

     • Conjugate graident methods, Long Chen, November 2020

     • Conjugate direction methods, Amir Ali Ahmadi, 2014

     • Conjugate gradient descent, Arti Singh

     • A  brief introduction to conjugate gradient method, Runar Heggelien Refsnaes, 2009

     • The conjugate gradient algorithm, University of Washington

     • Conjugate gradient, Holden Lee, 2020

  2.  Taylor series

     • On the remainder in the Taylor theorem, Lior Bary-Soroker, Eli Leher, 2008 

  3.   Neural Networks

     • Learning representations by back-propagating errors

Textbooks and Lecture Notes

Primary

• 1. An Introduction to Optimization with applications to Machine Learning, Edwin Chong. Wu-Sheng Lu, and Stanislaw Zak, Wiley, 5th edition.; free online (older) version, Amazon

  2. Linear Algebra and Optimization for Machine Learning, Charu Aggarawl, 2024

       Secondary

• 3. The mathematics for machine learning, Diesenroth, Faisal, and Ong

  4. Understanding Machine Learning: From Theory to Algorithms; Shai Shalev-Shwartz and Shai Ben-David; free online version;  Amazon

  5. Deep Learning, Ian Goodfellow, Yoshua Bengio and Aaron Courville,  Amazon

  6. Calculus and optimization

     1. Calculus, Linear Algebra, and Differential Forms,  Hubbard and Hubbard; online version (perhaps an older edition)   Amazon, Cornell website

     2. Advanced Calculus, Loomis and Sternberg

  7. Vector Calculus, Marsden and TrombaAn Introduction to Optimization, Edwin Chong and Stanislaw Zak, Wiley, 4th edition.

  8. The Science of Deep Learning by Iddo Drori,  Parts of the book is available at the author's website

  9. Patterns, Predictions, and Actions (mlstory.org) by Mortiz Hardt and Benjamin Recht; Amazon

  10. Geometry of Deep Learning: A Signal Processing Perspective by Jong Chul Ye; Amazon

  11. Convex Optimization - Algorithms and Complexity by Sebastien Bubeck; PDF from author's web page

  12. Dive into Deep Learning — Dive into Deep Learning 1.0.3 documentation (d2l.ai) by Aston Zhang and Zachary Lipton

  13. Neural Networks and Deep Learning: A Textbook by Charu Aggarwal; Amazon

  14. Neural networks and deep learning by Michael Nielsen

  15. Linear Algebra and Learning from Data (mit.edu) by Strang

  16. Neural Network Learning (cambridge.org)  by Anthony and Bartlett

  17. Optimization and Randomization

Supplementary Reading

1. NatureDeepReview.pdf (toronto.edu) 

2. Andrej Karpathy blog