15 – Meta-Learning - Hugo Larochelle (14/04/2021)

FLIPPED CLASS -- PLEASE VIEW THE VIDEO BEFORE CLASS

A lot of the recent progress on many AI tasks was enable in part by the availability of large quantities of labeled data. Yet, humans are able to learn concepts from as little as a handful of examples. Meta-learning is a very promising framework for addressing the problem of generalizing from small amounts of data, known as few-shot learning. In meta-learning, our model is itself a learning algorithm: it takes as input a training set and outputs a classifier. For few-shot learning, it is (meta-)trained directly to produce classifiers with good generalization performance for problems with very little labeled data. In this talk, Hugo present an overview of the recent research that has made exciting progress on this topic (including my own) and, if time permits, will discuss the challenges as well as research opportunities that remain.

Video: Hugo's Lecture (from last year's course) and the discussion from this lecture

Slides: Meta-Learning slides

13 – Self-Supervised Learning (07/04/2021)

In this lecture, Christos Tsirigotis will discuss self-supervised learning. We will discuss how to create representation beyond the supervised pre-training paradigm, and we are going to see how effective pretext tasks can be designed and how to train with contrastive objectives.

Slides:

• SSL slides

Video of SSL lecture.

Reference:

• Doersch, Carl, Abhinav Gupta, and Alexei A. Efros. "Unsupervised visual representation learning by context prediction." CVPR (2015).

• Gidaris, Spyros, Praveer Singh, and Nikos Komodakis. "Unsupervised representation learning by predicting image rotations." ICLR (2018).

• Wu, Zhirong, et al. "Unsupervised feature learning via non-parametric instance discrimination." CVPR (2018).

• He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." CVPR (2020).

• Chen, Ting, et al. "Big self-supervised models are strong semi-supervised learners." (2020).

• Grill, Jean-Bastien, et al. "Bootstrap your own latent: A new approach to self-supervised learning." NeurIPS (2020).

11 – Variational Autoencoders (22/03/2021-24/03/2021)

In this lecture, we will discuss a family of latent variable models known as the Variational Autoencoders (VAE). We’ll see how a deep latent gaussian model can be seen as an autoencoder via amortized variational inference, and how such an autoencoder can be used as a generative model. At the end, we’ll take a look at variants of VAE and different ways to improve inference.

Slides:

• Variational Autoencoders

Video of part I of Variational Autoencoders lecture.

Video of part II of Variational Autoencoders lecture.

Reference: (* = you are responsible for this material)

• *Chapter 20.10.3 of the Deep Learning textbook.

• *Chapter 2 of An Introduction to Variational Autoencoders by Kingma and Welling

• Inference Suboptimality in Variational Autoencoders by Chris Cremer (ICML 2018)

• Importance Weighted Autoencoders by Yuri Burda (ICLR 2016)

• Variational Inference, lecture note by David Blei. Section 1-6.

• Blog post Variational Autoencoder Explained by Goker Erdogan

• Blog post Families of Generative Models by Andre Cianflone

09 – Autoencoders and Autoregressive Generative Models (15/03/2021-17/03/2021)

In this lecture we will take a closer look at a form of neural network known as an Autoencoder. We will also begin our look at generative models with Autoregressive Models.

Slides:

• Autoencoders

• Autoregressive Generative Models

Video of Autoencoders lecture.

Video of part I of Autoregressive Generative Models lecture.

Video of part II of Autoregressive Generative Models lecture.

Reference: (* = you are responsible for this material)

• *Chapter 13-14 of the Deep Learning textbook.

• *Sections 20.10.5-20.10.10 of the Deep Learning textbook.

• The Neural Autoregressive Distribution Estimator by Hugo Larochelle and Iain Murray (AISTAT2011)

• MADE: Masked Autoencoder for Distribution Estimation by Mathieu Germain, Karol Gregor, Iain Murray, Hugo Larochelle (ICML2015).

• Pixel Recurrent Neural Networks by Aaron van den Oord, Nal Kalchbrenner, Koray Kavukcuoglu (ICML2016)

• *Conditional Image Generation with PixelCNN Decoders by Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, Koray Kavukcuoglu (NIPS2016)

07 – Optimization and Normalization Methods (24/02/2021)

In this lecture, we will discussion both popular and practical first-order optimization methods. We will not discuss but I do provide slides for some second-order methods and their interpretation.

Slides:

• Optimization I: First-Order Methods (and normalization methods)

• Optimization II: Second-Order Methods (optional)

Video of this lecture (24/02/2021)

Reference: (* = you are responsible for this material)

• *Chapter 8 of the Deep Learning textbook.

• Why Momentum Really Works. Gabriel Goh, Distill 2017.

05 – Sequential Models (15/02/2021, 17/02/2021)

In this lecture we introduce Recurrent Neural Networks and related models.

Lecture 05 RNNs (slides derived from Hugo Larochelle)

Video of this lecture, part I (15/02/2021)

Video of this lecture, part II (17/02/2021)

Reference: (* = you are responsible for this material)

• *Chapter 10 of the Deep Learning textbook (sections. 10.1-10.11, we will cover the material in 10.12 later).

• Blog post on Understanding LSTM Networks by Chris Olah.

03 – ConvNets I and II (01/02/2021, 08/02/2021 and 10/02/2021)

In this lecture we finish up our discussion of training neural networks and we introduce Convolutional Neural Networks.

Lecture 03 CNNs (some slides are modified from Hugo Larochelle’s course notes)

Backprop in CNNs (Slides are from Hiroshi Kuwajima’s Memo on Backpropagation in Convolutional Neural Networks.) -- I won't go over these in class, but they are required reading and will be the basis of one of the questions in Assignment 1.

Video of this lecture, part I (01/02/2021). note that it cuts off rather abruptly when I lost power at home!

Video_(a) and Video_(b) of this lecture, part II (08/02/2021) - in two parts to do technical difficulties

Video of this lecture, part III (10/02/2021).

Reference: (* = you are responsible for all of this material)

• *Chapter 9 of the Deep Learning textbook, Sections 9.10 and 9.11 are optional.

• Andrej Karpathy’s excellent tutorial on CNNs.

• Paper on convolution arithmetic by Vincent Dumoulin and Francesco Visin.

• WaveNet Blog presenting dilated convolutions animation and samples.

• Blog on Deconvolution and Checkerboard Artifacts by Augustus Odena, Vincent Dumoulin and Chris Olah.