Deep Learning (es667)

Aim

The aim of this course is to introduce students to deep learning techniques. Students will be taken from the fundamentals to the research level questions. Students will be able to get an overview of the most important architectures, as well as the principles behind training  such models. We will also cover specific important applications from various scientific domains. In order to get the most out ot the course, students needs to be mathematically motivated as well as comfortable in coding, preferably in python. 

Prequisites

There are no official pre-requisites for this course but it would help if you have covered the following topics in some courses. The links provided are also very good sources. 

1. Linear algebra [Online course from MIT] 

2. Probability [Online course from MIT] 

3. Calculus [Online course from MIT]

4. Preferably machine learning course at IITGN (ES 335).

Contents

• History for DL. Standard architectures e.g. multilayer perception.

• CNNs, RNNs, modern RNNs, attention mechanism

• Optimization algorithms -- SGD, momentum , ADAM, second-order optimization methods 

• Applications in NLP, computer vision

• Graph neural networks, connection to Weisfeiler-Leman algorithm

• Transformers and few architectures e.g. BERT, RoBERTa, Distill-BERT

• Efficient transformer architectures using random projection and low-rank approximations e.g. Reformer, Hashformer

• Learning theory -- VC dimension, Rademacher complexity, and generalization bounds. Learning theory in the context of deep learning. 

• Generative adversarial networks

• Gaussian processes, NTKs

• Hyperparameter optimization

• Reinforcement learning

Logistics

1. Instructor -- Anirban Dasgupta

2. TA -- Shrutimoy Das, Brian Mwigo.

3. Lectures -- as per timetable.

Grading policy is roughly as follows. Will be finalized in the first couple of lectures.

• Assessment 1, 2 : 20 + 30 = 50%

• Homeworks: 20% (will primarily involve coding)

• Project: 30% . Will have components of regular updates, presentations and report writing.  We will share topics for your project ideas. 

We will strictly follow the honor policy of IITGN. Collaboration in homeworks is allowed unless stated explicitly.  Everyone needs to write down their own answers and code as well as generate their own plots. Anyone you discuss ideas with when solving the homework needs to be acknowledged clearly. You are not expected to use Google or any other source for finding answers to homework questions unless explicitly allowed. 

Textbooks and References:

1. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. An MIT Press book. 2016.

2. Ashton Zhang, Zachary Lipton, Dive into Deep Learning

3. Understanding Deep learning, Simon Prince. 

4. "Deep Learning - A Visual Approach" by Andrew Glassner.

5. Notes by Matt Telgarsky on theory of deep learning. 

 Similar Courses.

• https://people.csail.mit.edu/moitra/408c.html 

• http://cse.iitm.ac.in/~miteshk/CS6910.html 

• https://inst.eecs.berkeley.edu/~cs182/sp23/ 

• http://cs231n.stanford.edu/