Pattern Recognition and Machine Learning

Course Information

Space: CRC 102

Time: E Slot. (Tue: 11AM, Wed: 10AM, Thu: 8AM, Friday: 4:50 PM)

Teaching Assistants: TBA

Learning Outcomes

At the end of the course, the student should be able:

1. To understand the use cases and limitations of machine learning.

2. To recognise the type of learning problem suitable for a practical task at hand.

3. To identify the data required for solving a given type of learning problem.

4. To identify the right learning algorithms for solving a given learning problem. [KEY]

5. To analyse (several) learning algorithms and identify the role of the various critical knobs in the algorithms. [KEY]

6. To efficiently use various software packages for solving learning problems.

7. To implement various learning algorithms from first principles. [KEY]

Grading

• Mid-term: 20%

• Final exam: 30%

• Min-Quizzes: 15% (Best 3 out of 4)

• Programming Assignments: 20 (10 for each assignment)

• Programming Contest: 15%

Important Dates

Mini-Quiz 1: Feb 1

Mini-Quiz 2: Feb 22

Midsem: Mar 8

Mini-Quiz 3:Mar 22

Mini-Quiz 4: Apr 12

Assignment 1: Available on Feb 8. Due on March 1.

Assignment 2: Available on Mar 15. Due on April 12.

Data contest: Due April 26.

Reference Books:

• [DHS] Richard O. Duda, Peter E. Hart and David G. Stork. Pattern Classification. John Wiley, 2001.

• [CB] Christopher M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

  • Download PDF from here : https://www.microsoft.com/en-us/research/people/cmbishop/#!prml-book

• [SB] Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014.

  • Download PDF from here: http://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/copy.html

• [JW] Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani. An Introduction to Statistical Learning with Applications in R, Springer, 2014.

  •  Download PDF from here: http://www-bcf.usc.edu/~gareth/ISL

  • Jupyter notebooks for ISLR: https://github.com/JWarmenhoven/ISLR-python

Related Resources on the Web:

• Google crash course on machine learning: https://developers.google.com/machine-learning/crash-course/ml-intro

• Intel course on Machine learning: https://software.intel.com/en-us/ai-academy/students/kits/machine-learning-501

• Nptel course on Machine learning: https://nptel.ac.in/downloads/117108048/

• Hal Daume III. A course in machine learning. http://ciml.info/

Reference material for Pre-requisites:

1. Undergraduate multivariate calculus.

1a. https://ocw.mit.edu/courses/mathematics/18-01sc-single-variable-calculus-fall-2010/unit-2-applications-of-differentiation/ (Part A)

1b. https://ocw.mit.edu/courses/mathematics/18-02sc-multivariable-calculus-fall-2010/2.-partial-derivatives/ (Part A and B)

2. Linear algebra. 

2a. https://youtu.be/kjBOesZCoqc (Essence of linear algebra, youtube series)

2b. https://youtu.be/ZK3O402wf1c (Gilbert Strang course on Linear algebra.)

Books:

Gilbert Strang. Linear Algebra and its Applications.

3. Probability.

https://ocw.mit.edu/resources/res-6-012-introduction-to-probability-spring-2018/part-i-the-fundamentals/

Books:

Bertsekas and Tsitsiklis. Introduction to Probability.

4. Basic Python/Numpy programming. 

https://developers.google.com/edu/python/

http://www.astro.up.pt/~sousasag/Python_For_Astronomers/Python_qr.pdf

http://cs231n.github.io/python-numpy-tutorial/

https://www.tutorialspoint.com/numpy

Announcements:

1. Mini-quiz 1 to be held on Feb 1

2. Programming Assignment 1 released on Feb 12 in Moodle. Part A due on March 1 11:55 PM.

3. New worksheet released on Feb 17.

4. Programming assignment 1 Part  A, deadline extended to Sunday March 3, 11:55 PM.

5. Mid-sem exam on March 8 Friday.

6. Programming Assignment 2 released. Due March 29.

7. Data contest released. Due April 26.

8. Programming Assignment 3 released. Due April 26.

Supporting Material:

Class Notes/Slides:

 Course instructional handout : pdf

1.  Introduction slides : Jan 16 : pdf

2.  Calculus/Geometry notes : Jan 17 : pdf

3.  Probability/Linear algebra/optimisation notes: Jan 18-25 : pdf

4.  Bayes classifier, learning from data : Jan 25-31 : pdf

5.  Regression : Feb5-13 : pdf

6.  Logistic Regression : Feb 14-19 : pdf

7. Constrained Optimisation : Feb 19 -21 : pdf

8. SVM : Feb 27-Mar 1 : pdf

9. Decision Trees and Boosting: Mar 6 - 13 : pdf

10. Ensemble methods: Bagging, Random Forests and Boosting : Mar 7-13 :  Slides pdf

11. Classification wrap up: kNN, ANN, Multiclass, Surrogate loss functions, Structured prediction : Mar 19-26 : pdf

12. PCA : pdf

13. Clustering: slides

14. Collaborative filtering: pdf

15. EM algorithm : Bishop Chapter 9

16. Regularisation: TBA

17. Learning theory : pdf

Worksheets:

1. Probability basics : Available on Moodle

2. Mega-worksheet - Prerequisite/Regression/classification : pdf

3. Linear algebra/least squares regression comprehensive worksheet: Gilbert Strang (4th Ed) book exercises from Sections 3.1,  3.2 and 3.3.

4. Classification worksheet: Gather from "Exercises" in the notes.

5. Multiclass, clustering, PCA, collaborative filtering: pdf

6. EM algorithm : pdf

Quiz papers/Solutions:

1. Mini-quiz 1: Qn paper : Solutions 

2. Mini-quiz 2: Qn paper : Solutions

3. Mid-sem : Qn paper : Solutions

4. Mini-quiz 3: Qn paper : Solutions

5. Mini-quiz 4: Qn paper : Solutions

Jupyter Notebooks:

1. Multivariate-Normal illustration : notebook

2. 1d-Regression illustration :  notebook

3. Programming assignment 1: Released on Moodle.

4. SVM illustration: notebook

5. Programming Assignment 2: Released on Moodle.

6. Programming Assignment 3: Released on Moodle

7. Lasso Regression illustration: notebook