Deep Learning (ES667)

Aim

The aim of this course is to introduce students to deep learning techniques. Students will be taken from the fundamentals to the research level questions. Students will be able to get an overview of the most important architectures, as well as the principles behind training  such models. We will also cover specific important applications from various scientific domains. In order to get the most out ot the course, students needs to be mathematically motivated as well as comfortable in coding, preferably in python. 

Prequisites

There are no official pre-requisites for this course but it would help if you have covered the following topics in some courses. The links provided are also very good sources. 

1. Linear algebra [Online course from MIT] 

2. Probability [Online course from MIT] 

3. Calculus [Online course from MIT]

4. Preferably machine learning course at IITGN (ES 335).

Contents

• History for DL. Standard architectures e.g. multilayer perception, universal approximation theorem. 

• CNNs and applications

• Optimization algorithms -- SGD, momentum , ADAM, second-order optimization methods 

• Graph neural networks, connection to Weisfeiler-Leman algorithm

• Attention mechanism, transformers and few architectures e.g. BERT, RoBERTa, Distill-BERT

• Efficient transformer architectures using random projection and low-rank approximations e.g. Reformer, Hashformer

• Learning theory -- VC dimension, Rademacher complexity, and generalization bounds. Learning theory in the context of deep learning. 

• Generative models -- Generative adversarial networks, normalizing flows

• Hyperparameter optimization

• Reinforcement learning

Logistics 

1. Instructor -- Anirban Dasgupta 

2. TA --  Binita Maity, Manavendra Singh.

3. Lectures -- as per timetable.

Grading policy is roughly as follows. Will be finalized in the first couple of lectures.

• Assessment 1, 2 : 20 + 30 = 50%

• Homeworks: 20% (will primarily involve coding)

• Project: 30% . Will have components of regular updates, presentations and report writing.  We will share topics for your project ideas. 

We will strictly follow the honor policy of IITGN. Collaboration in homeworks is allowed unless stated explicitly.  Everyone needs to write down their own answers and code as well as generate their own plots. Anyone you discuss ideas with when solving the homework needs to be acknowledged clearly. You are not expected to use Google or any other source for finding answers to homework questions unless explicitly allowed. 

Textbooks and References:

1. Understanding Deep learning, Simon Prince. 

2. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. An MIT Press book. 2016.

3. Ashton Zhang, Zachary Lipton, Dive into Deep Learning

4. "Deep Learning - A Visual Approach" by Andrew Glassner.

5. Alice’s Adventures in a differentiable wonderland by Simone Scaradapane. 

6. Notes by Matt Telgarsky on theory of deep learning. 

 Similar Courses.

• https://people.csail.mit.edu/moitra/408c.html 

• http://cse.iitm.ac.in/~miteshk/CS6910.html 

• https://inst.eecs.berkeley.edu/~cs182/sp23/ 

• http://cs231n.stanford.edu/

Scribe Notes.

• January 07, 2025  -- introduction

• January 09, 2025  -- activation functions, difference between linear and nonlinear predictors

• January 14, 2025  -- shallow neural networks, universal approximation theorem.

• January 16, 2025  -- shallow neural networks contd.

• January 23, 2025  -- gradient descent

• January 30, 2025  -- SGD, SGD with momentum

• February 04, 2025 -- regularization

• February 11, 2025  -- convolution, CNNs

• February 13, 2025 -- autoencoders

• February 18, 2025  -- understanding CNNs

• February 20, 2025  -- graph representation learning, random walks

• February 27, 2025 

• Extra 1