15 – Meta-Learning by Hugo Larochelle (26th November 2021)

FLIPPED CLASS -- PLEASE VIEW THE VIDEO BEFORE CLASS

In this lecture, Hugo will answer questions from students about Meta-learning. Students are supposed to watch the below video before class.

In-class recorded lecture
(See Piazza for passcode)

Hugo's lecture video (watch before class) -- this lecture was recorded as part of IFT6135-H2020
Hugo's lecture slides (pdf)

Reference:

• See slides for references to relevant papers.

13 – Self-Supervised Learning (19th November 2021)

In this lecture, we will discuss self-supervised learning. We will discuss how to create representation beyond the supervised pre-training paradigm, and we are going to see how effective pretext tasks can be designed and how to train with contrastive objectives.

Recorded Lecture
(See Piazza for passcode)

Slides
(Most slides are from IFT6135 - H2021 taught by Aaron Courville)

Reference:

• Doersch, Carl, Abhinav Gupta, and Alexei A. Efros. "Unsupervised visual representation learning by context prediction." CVPR (2015).

• Gidaris, Spyros, Praveer Singh, and Nikos Komodakis. "Unsupervised representation learning by predicting image rotations." ICLR (2018).

• Wu, Zhirong, et al. "Unsupervised feature learning via non-parametric instance discrimination." CVPR (2018).

• He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." CVPR (2020).

• Chen, Ting, et al. "Big self-supervised models are strong semi-supervised learners." (2020).

• Grill, Jean-Bastien, et al. "Bootstrap your own latent: A new approach to self-supervised learning." NeurIPS (2020).

11 – Variational Autoencoders (5th and 9th November 2021)

In this lecture, we will discuss a family of latent variable models known as the Variational Autoencoders (VAE). We’ll see how a deep latent gaussian model can be seen as an autoencoder via amortized variational inference, and how such an autoencoder can be used as a generative model. At the end, we’ll take a look at variants of VAE and different ways to improve inference.

Recorded Lecture (Part I) (from 55.48 until the end)
Recorded Lecture (Part II)
(See Piazza for passcode)

Slides (pdf), Slides (key)
(Most slides are from IFT6135 - H2021 taught by Aaron Courville)

Reference:

• Chapter 20.10.3 of the Deep Learning textbook.

• Chapter 2 of An Introduction to Variational Autoencoders by Kingma and Welling

• Inference Suboptimality in Variational Autoencoders by Chris Cremer (ICML 2018)

• Importance Weighted Autoencoders by Yuri Burda (ICLR 2016)

• Variational Inference, lecture note by David Blei. Section 1-6.

• Blog post Variational Autoencoder Explained by Goker Erdogan

• Blog post Families of Generative Models by Andre Cianflone

09 – Autoencoders and Autoregressive Generative Models (26th October and 2nd November 2021)

Introduction to autoencoders and different types of autoregressive generative models.

Recorded Lecture (Part I)
Recorded Lecture (Part II) (from the start until 54.44)
(See Piazza for passcodes)

Autoencoders -- Slides (pdf), Slides (key)
Autoregressive Generative Models -- Slides (pdf), Slides (key)
(Most slides are from IFT6135 - H2021 taught by Aaron Courville)

Reference:

• Chapter 13-14 of the Deep Learning textbook.

• Sections 20.10.5-20.10.10 of the Deep Learning textbook.

• The Neural Autoregressive Distribution Estimator by Hugo Larochelle and Iain Murray (AISTAT2011)

• MADE: Masked Autoencoder for Distribution Estimation by Mathieu Germain, Karol Gregor, Iain Murray, Hugo Larochelle (ICML2015).

• Pixel Recurrent Neural Networks by Aaron van den Oord, Nal Kalchbrenner, Koray Kavukcuoglu (ICML2016)

• Conditional Image Generation with PixelCNN Decoders by Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, Koray Kavukcuoglu (NIPS2016)

07 – Optimization and Normalization Methods (8th October 2021)

Discussion on popular and practical first-order optimization methods and discussion on normalization methods. We will not discuss second-order optimization methods but slides are provided below for your own reading (optional).

Recorded Lecture
(See Piazza for passcode)

Slides (pdf), Slides (key)
(Optional) Slides for second-order optimization methods
(Most slides are from IFT6135 - H2021 taught by Aaron Courville)

Reference:

• Chapter 8 of the Deep Learning textbook.

• Why Momentum Really Works. Gabriel Goh, Distill 2017.

05 – Sequential Models (24th and 28th September 2021)

Introduction to Recurrent Neural Networks and related models.

Recorded Lecture (Part I) (from 1.09.15 until the end)
Recorded Lecture (Part II)
(See Piazza for passcodes)

Slides (pdf), Slides (key) (Most slides are from IFT6135 - H2021 taught by Aaron Courville)

Reference:

• Chapter 10 of the Deep Learning textbook (sections. 10.1-10.11)

• Blog post on Understanding LSTM Networks by Chris Olah

03 – Convolutional Neural Networks (14th, 17th and 24th September 2021)

Introduction to Convolutional Neural Networks.

Recorded Lecture (Part I)
Recorded Lecture (Part II)
Recorded Lecture (Part III) (from the start until 59.10)
(See Piazza for passcodes)

Slides (pdf), Slides (key) (Most slides are from IFT6135 - H2021 taught by Aaron Courville)

Backprop in CNNs (Slides are from Hiroshi Kuwajima’s Memo on Backpropagation in Convolutional Neural Networks.) -- We won't go over these in class, but they are required reading and will be the basis of one of the questions in Assignment 1.

Reference:

• Chapter 9 of the Deep Learning textbook, Sections 9.10 and 9.11 are optional.

• Andrej Karpathy’s excellent tutorial on CNNs.

• Paper on convolution arithmetic by Vincent Dumoulin and Francesco Visin.

• WaveNet Blog presenting dilated convolutions animation and samples.

• Blog on Deconvolution and Checkerboard Artifacts by Augustus Odena, Vincent Dumoulin and Chris Olah.

01 – Introduction to Neural Networks (7th September 2021)

Beginning of detailed introduction to neural networks.

Recorded Lecture (See Piazza for passcode)
Slides (pdf), Slides (key) (Most slides are from IFT6135 - H2021 taught by Aaron Courville)

Reference:

• Chapter 6 of the Deep Learning textbook (by Ian Goodfellow, Yoshua Bengio and Aaron Courville).