14 – Self-Supervised Learning

In this lecture, we will discuss self-supervised learning. We will discuss how to create representation beyond the supervised pre-training paradigm, and we are going to see how effective pretext tasks can be designed and how to train with contrastive objectives.

Guest lecture by Olivier J. Hénaff
Lecture Slides: PDF Google slides

Reference: 

• See slides for references to relevant papers.

12 – Generative Adversarial Networks

In this lecture, we will discuss Generative Adversarial Networks (GANs). GANs are a recent and very popular generative model paradigm. We will discuss the GAN formalism, some theory and practical considerations.

Lecture Slides: PDF  Keynote
(slides built on top of Aaron Courville's course slides)

Reference: 

• Sections 20.10.4 of the Deep Learning textbook.

• Generative Adversarial Networks by Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio (NIPS 2014).

• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization by Sebastian Nowozin, Botond Cseke and Ryota Tomioka (NIPS 2016).

• NIPS 2016 Tutorial: Generative Adversarial Networks by Ian Goodfellow, arXiv:1701.00160v1, 2016

• Adversarially Learned Inference by Vincent Dumoulin , Ishmael Belghazi , Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky and Aaron Courville (ICLR 2017).

• Many others references in the slides.

10 –  Variational Autoencoders

In this lecture, we will discuss a family of latent variable models known as the Variational Autoencoders (VAE). We’ll see how a deep latent gaussian model can be seen as an autoencoder via amortized variational inference, and how such an autoencoder can be used as a generative model. At the end, we’ll take a look at variants of VAE and different ways to improve inference.

Lecture Slides: PDF  Keynote
(slides built on top of Aaron Courville's course slides)

Reference: 

• Chapter 20.10.3 of the Deep Learning textbook.

• Chapter 2 of An Introduction to Variational Autoencoders by Kingma and Welling

• Inference Suboptimality in Variational Autoencoders by Chris Cremer (ICML 2018)

• Importance Weighted Autoencoders by Yuri Burda (ICLR 2016)

• Variational Inference, lecture note by David Blei. Section 1-6.

• Blog post Variational Autoencoder Explained by Goker Erdogan

• Blog post Families of Generative Models by Andre Cianflone

08 – Regularization Methods

Detailed discussion of regularization methods and their interpretation.

Lecture Slides: 
Part I: Parameter Regularization: PDF  Keynote
Part II: Dropout:  PDF  Keynote
(slides built on top of Aaron Courville's course slides)

Reference: 

• Chapter 7 of the Deep Learning textbook.

• Understanding deep learning requires rethinking generalization (ICLR 2017) by Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals

06 – Self-Attention and Transformer

Introduction to Transformer based models. 

Lecture Slides:  PDF  Keynote (slides built on top of Aaron Courville's course slides)

Reference: 

• Attention is all you need, The Illustrated Transformer,

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,

• GPT, GPT-2, GPT-3, The Illustrated GPT-2,

• Better Language Models and Their Implications, 

• Image Transformer, Vision Transformer,

• Music Transformer.

04 – PyTorch Tutorial

This lecture will be an introductory tutorial to PyTorch. A Colab notebook will be provided for the tutorial. You are encouraged to bring your laptop to the class and follow along.

Material: Colab Notebook

This tutorial will cover

• the torch.Tensor class, and important attributes and operations

• automatic differentiation in pytorch

• torch.nn and torch.optim modules

• training MLPs and ConvNets on MNIST

02 – Training Neural Networks

Detailed discussion on how to train a feedforward neural network.

Lecture Slides:  PDF  Keynote (slides built on top of Aaron Courville's course slides)

Reference: 

• Chapter 6 of the Deep Learning textbook (by Ian Goodfellow, Yoshua Bengio and Aaron Courville).

00 – Overview of Deep Learning and Background Material

Provide an overview of latest developments in Deep Learning, what Deep Learning is and highlight some foundational material which serve as prerequisites for this course: linear algebra, calculus, and the basics of machine learning.

Lecture Slides:  PDF  Keynote (slides built on top of Aaron Courville's course slides)
Reference:

• Chapters 1-5 of the Deep Learning textbook (by Ian Goodfellow, Yoshua Bengio and Aaron Courville).