IFT 6268 - Self-supervised Representation Learning

Fall 2020, A Course offered by the Université de Montréal

IFT6268-A2020 DIRO, Université de Montréal

Course Description:

There has been a great deal of recent progress on representation learning. Most of it coming in the form of so-called self-supervised representation learning. In this course, we will take a fairly broad interpretation of what constitutes self-supervised methods and include some unsupervised and supervised learning methods when appropriate. We are interested in methods that learn meaningful and effective semantic representations without (exclusively) relying on labeled data. More specifically, we will be interested in methods such as: Data augmentation tasks, Knowledge distillation, Self-distillation, Iterative learning, Contrastive methods (DIM, CPC, MoCo, SimCLR, etc), BYOL, and the analysis of self-supervised methods.

Our goal will be to gain an understanding how these methods work and what fundamental principles are at play.

This is an advanced seminar course on the topic and as such, we will read and discuss a wide array of recent and classic papers. Lectures will be largely student lead. We assume knowledge of the fundamentals of machine learning (and in particular deep learning -- as you would find in IFT6135), we will also explore applications of self-supervised representation learning to a wide range of domains, including Natural Language Processing, Computer Vision and Reinforcement Learning. Basic knowledge of these domains will also be assumed.

Student evaluation will be done based on two components: 10% class participation; 40% for class presentations (2 per student); and 50% for a course project.

Course Material:

In this course, we will take a very wide scope on self-supervised learning (SSL) in general and deep learning in particular. Deep learning has recently been responsible for a large number of impressive empirical gains across a wide array of applications including most dramatically in object recognition and detection in images and speech recognition.

In this course we will explore the recent advances in the area of representation learning. Through student lead seminars, we will review the recent literature with a eye toward establishing

In particular topics covered by the course include the following:

  • Engineering tasks for Computer Vision

  • Contrastive learning methods

  • Generative Methods

  • Bootstrap Your Own Latents (BYoL)

  • Self-distillation Methods

  • Self-training / Pseudo-labeling Methods

  • SSL for Natural Language Processing

  • Iterated Learning / Emergence of Compositional Structure

  • SSL for Video / Multi-modal data

  • The role of noise in representation learning

  • SSL for RL, control and planning

  • Analysis of Self-Supervised Methods

  • Theory of SSL

  • Unsupervised Domain Adaptation

Instruction style: This is a seminar course. The vast majority of the lecture time will be devoted to (i) students presenting papers to each other; and (ii) group discussion of the papers.

Workload:

This is something that I've been asked about. The expectation for each student is that you will read 4 papers / week, submit 1 question each. Twice in the term, each student (with a partner) will present a papers and lead the class discussion on that paper. Finally you are expected to complete a course project using a method related to the material seen in class. I expect the course project will take up the majority of your time spent work on course related material.

Evaluation:

As a seminar course the evaluation is based on the following:

    1. (40% of total grade) Lectures based on recent papers in the literature. Students (in groups of 2) present 2, 1h-long lectures. The goal is to present a paper or a set of papers on a narrow topic. The presentation should include a novel synthesis of the material. Slides and lecture notes are to be made available on the course website. The presentation of novel experiments exploring and reproducing the methods is encouraged but not strictly required. Lectures should include the following:

      • A summary of the key ideas, motivations and insights of the work

      • A contextualization of the work within the literature - especially the literature that, at the point of presentation, that was previously reviewed in the course.

      • Answering a curated set of submitted questions from students.

      • (Update Sept. 22, 2020) Feedback from after the first set of talks: We've had a great set of early talks, but there are a few points that are worth keeping in mind when building your next talk:

        1. Keep in mind that the class has (or should have) all already read the assigned paper. It's important to present the paper but the emphasis should be on providing context and either diving more thoroughly into specific aspects of the paper or present follow-on work. You can also briefly talk about your own ideas of how to build upon the paper for future work.

        2. When going beyond the assigned papers, here you need to slow-down and provide context for the class. They likely aren't aware of this work and you can't present it at the same pace as you would present the assigned paper.

        3. Use lots of illustrations and other visual aids. In particular, don't limit yourself to the figures and illustrations that are found in the assigned paper.

        4. Whenever possible, try to open up the lecture for discussions where other students can pitch in on an important point.

        5. Try to pick just a few questions from the slack channels (maybe 3-5) and go deep in answering them. If the answer is you don't know, then maybe skip that question or alternatively open it up to discussion for the class. Also try to pause a couple of times in between the lecture to take questions / comments.

    2. (50% of total grade) Students will work on a term-long course project.

    3. (10% of total grade) Participation. As a seminar course, your presentation is key to a great learning experience for all students. These participation points are there to reflect this reality. Elements of engagement include: asking questions in class to the presenters. Engaging on the class forum (asking and answering questions). This is not meant to be a "make-work" element for students. If you are engaged in the course, these should be easy points to get.

      • For each paper assigned for reading, each student is to submit a question.

Instructors: