IFT 6268 - Self-supervised Representation Learning
Fall 2020, A Course offered by the Université de Montréal
Lecture 1 (01/09/2020) : Introduction I - What is self-supervised representation learning?
Lecturer: Aaron Courville
Slides: Introduction to self-supervised representation learning
Video: here. Passcode: J#HvN2+c
Lecture 2 (03/09/2020) : Introduction II - Overview of self-supervised representation learning?
Lecturer: Aaron Courville
Slides: Overview of self-supervised representation learning
Video: here Passcode: Sj0p24#s
Lecture 3 (08/09/2020)
Paper I: Dosovitskiy et al. (2014) Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks. IEEE Trans. Pattern Analysis and Machine Intelligence
Lecturers: Nitarshan Rajkumar and Christos Tsirigotis
Notes: Exemplar CNN (HackMD)
Video: here
Paper II: Doersch et al (2015) Unsupervised Visual Representation Learning by Context Prediction ICCV 2015
Lecturer: Tianyu Zhang and Eeshan Dhekane
Slides: Context Prediction (Google Slides)
Video: here
Lecture 4 (10/09/2020)
Paper I: Gidaris, Singh and Komodakis (2018) Unsupervised Representation Learning by Predicting Image Rotations ICLR 2018
Lecturers: Harmanpreet Singh and Rana Akshay Singh
Slides: Rotations (GDrive)
Video: here
Paper II: Kolesnikov, Zhai and Beyer (2019) Revisiting Self-Supervised Visual Representation Learning. CVPR 2019
Lecturers: Makesh Narsimhan and Francois Mercier
Slides: Revisiting SSL
Video: here
Lecture 5 (15/09/2020)
Paper I: Zhang et al (2016) Colorful Image Colorization ECCV 2016
Lecturers: Arnaud L'Heureux and Leila Fabing
Slides: Colorization (Google Drive)
Video: here
Paper II: Lee et al (2020) Predicting What You Already Know Helps: Provable Self-Supervised Learning ArXiv 2020
Lecturers: Yutong Yan and Qiwei Shao
Slides: Provable SSL (Google Slides)
Video: here
Lecture 6 (17/09/2020)
Paper I: Misra, Zitnick and Hebert (2016) Shuffle and Learn: Unsupervised Learning using Temporal Order Verification ECCV 2016
Lecturers: Yifan Bai (Andy) and Zhen Liu
Slides: Shuffle and Learn (Google Slides)
Video: here
Paper II: Wang and Gupta (2015) Unsupervised Learning of Visual Representations using Videos ICCV 2015
Lecturers: Maksym Perepichka and Afuad Hossain
Slides: Wang and Gupta (Google Slides)
Video: here
Lecture 7 (22/09/2020)
Paper I: Asano et al (2019) A critical analysis of self-supervision, or what we can learn from a single image ICLR 2020
Lecturers: Touraj laleh and Maziar Shahi
Slides: Mono SSL (Google Slides)
Paper II: Bojanowski and Joulin (2017 ) Unsupervised Learning by Predicting Noise. ICML 2017
Lecturers: Charles Lagacé and Ehsan Rezai
Slides: NAT (Google Slides)
Video for both lectures: here
Lecture 8 (24/09/2020)
Paper I: van den Oord et al. (2018) Representation Learning with Contrastive Predictive Coding (CPC), ArXiv 2018
Lecturers: Fanny Salvail-Bérard and Erfan Salehi
Slides: CPC (Google Slides)
Video: here
Paper II: Tian et al. (2019) Contrastive Multiview Coding (CMC) ArXiv 2019
Lecturers: Hattie Zhou and Rana Akshay Singh
Slides: CMC (Google Slides)
Video: here
Lecture 9 (29/09/2020)
Paper I: Chen T et al (2020) A Simple Framework for Contrastive Learning of Visual Representations (SimCLR, see also SimCLRv2). ICML 2020
Lecturers: Yan Coté and Shanel Gauthier
Slides: SimCLR (Google Drive)
Video: (not available ... sorry about that)
Paper II: He et al (2020) Momentum Contrast for Unsupervised Visual Representation Learning (MoCo, see also MoCo v2). CVPR 2020
Lecturers: Raghav Gupta and Melissa Mozifian
Slides: MoCo (HackMD)
Video: here
Lecture 10 (01/10/2020)
Paper I: Xiao et al (2020) What Should Not Be Contrastive in Contrastive Learning ArXiv 2020
Lecturers: Marc-André Ruel and Maxime Daigle
Slides: LooC (Google Slides)
Video: here
Paper II: Tschannen, et al (2019) On mutual information maximization for representation learning. ArXiv 2019.
Lecturers: Christos Tsirigotis and Eeshan Gunesh Dhekane
Slides: InfoMax (HackMD)
Video: here
Lecture 11 (06/10/2020)
Paper I: Hjelm and Bachman (2020) Representation Learning with Video Deep InfoMax. (VDIM) Arxiv 2020
Lecturers: Charles Lagacé
Slides: VDIM (Google Slides)
Video: here
Paper II: Chen et al (2020) Generative Pretraining from Pixels (iGPT) ICML 2020
Lecturers: Milad Aghajohari and Meraj Hashemizadeh
Slides: iGPT (Google Slides)
Video: here
Lecture 12 (08/10/2020)
Paper I: Grill et al (2020) Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (BYoL). ArXiv 2020
Lecturers: Nathaniel Simard and Guillaume Lagrange
Slides: BYOL (Google Slides)
Video: here
Paper II: Xie et al (2020) Self-training with Noisy Student improves ImageNet classification. CVPR 2020
Lecturers: Rahim Kassanaly and Nicolas Trudel-Mallet
Slides: Noisy Student (Google Slides)
Video: here
Lecture 13 (13/10/2020)
Paper I: Sohn and Berthelot et al. (2020) FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. ArXiv 2020
Lecturers: Milad Aghajohari and Meraj Hashemizadeh
Slides: FixMatch (Google Slides)
Video: here
Paper II: Müller et al (2019) When Does Label Smoothing Help? NeurIPS 2019
Lecturers: Hattie Zhou and Fanny Salvail-Bérard
Slides: Label Smoothing (Google Slides)
Video: here
Lecture 14 (15/10/2020)
Paper I: Zhang et al (2019) Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation ICCV 2019
Lecturers: Marc-Andre Ruel and Yan Coté
Slides: Be Your Own Teacher (Google Slides)
Video: here
Paper II: Devlin et al (2019) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. (BERT) NAACL 2019
Lecturers: Qiwei Shao and Nicolas Trudel-Mallet
Slides: BERT (Google Slides)
Video: here
Lecture 15 (27/10/2020)
Paper I: Brown et al (2020) Language Models are Few-Shot Learners (GPT-3 and Neural Scaling Laws) ArXiv 2020
Lecturers: Michael Noukhovitch and Ethan Caballero
Slides: GPT-N and Neural Scaling Laws (Google Slides)
Video: here
Paper II: Peters et al (2018) Deep contextualized word representations (ELMO), NAACL 2018
Lecturers: Rahim Kassanaly and Harmanpreet Singh
Slides: ELMO (Google Slides)
Video: here
Lecture 16 (29/10/2020)
Paper I: Lu et al (2019) ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks, NeurIPS 2019
Lecturers: Makesh Narsimhan and Ian Porada
Slides: ViLBERT (Google Drive)
Video: here
Paper II: Clark et al (2020) ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators ICLR 2020
Lecturers: Darshan Patil and Francois Mercier
Slides: ELECTRA (Google Slides)
Video: here
Lecture 17 (03/11/2020)
Paper I: Cogswell et al. (2020) Emergence of Compositional Language with Deep Generational Transmission ArXiv 2020
Lecturers: Darshan Patil and Ian Porada
Slides: Cultural Transmission (Google Slides)
Video: here
Paper II: Kharitonov and Baroni (2020) Emergent Language Generalization and Acquisition Speed are not tied to Compositionality
Lecturers: Max Schwarzer and Michael Noukhovitch
Slides: Compositionality (Google Slides)
Video: here
Lecture 18 (05/11/2020)
Paper I: Pathak et al. (2017) Curiosity-driven Exploration by Self-supervised Prediction (see also a large-scale follow-up) ICML 2017
Lecturers: Ethan Caballero and Melissa Mozifian
Slides: ICM (Google Slides)
Video: here
Paper II: Schwarzer, Anand et al. (2020) Data-Efficient Reinforcement Learning with Momentum Predictive Representations.
(note: title change)
Lecturers: Max Schwarzer and Nitarshan Rajkumar
Slides: SPR (Google Slides)
Video: here
Lecture 19 (10/11/2020)
Paper I: Sekar and Rybkin et al. (2020) Planning to Explore via Self-Supervised World Models. ICML 2020
Lecturers: Yutong Yan and Ehsan Rezaei
Slides: Plan2Explore (Google Slides)
Video: here
Paper II: Anand et al. (2019) Unsupervised State Representation Learning in Atari (ST-DIM) NeurIPS 2019
Lecturers: Erfan Salehi and Abderrahim Fathan
Slides: STDIM (Google Slides)
Video: here
Lecture 20 (12/11/2020)
Paper I: Vu et al. (2018) ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation CVPR 2019
Lecturers: Tianyu Zhang and Yifan Bai (Andy)
Slides: ADVENT (Google Slides)
Video: here
Paper II: Chen et al (2019) Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera
Lecturers: Simon Ramstedt and Pierre-André Brousseau
Slides: GLNet (Google Slides)
Video: here
Lecture 21 (17/11/2020)
Paper I: Ermolov et al. (2020) Whitening for Self-Supervised Representation Learning ArXiv
Lecturers: Afuad-Abrar Hossain and Maksym Perepichka
Slides: Whitening (Google Slides)
Video: here
Paper II: Misra and van der Maaten (2020) Self-Supervised Learning of Pretext-Invariant Representations. CVPR 2020
Lecturers: Pierre-André Brousseau and Leila Fabing
Slides: PiRL (Google Slides)
Video: here
Lecture 22 (19/11/2020)
Paper I: Tian et al. (2020) Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? ArXiv
Lecturers: Maziar M-Shahi and Touraj Laleh
Slides: SSL-FewShot (Google Slides)
Video: here
Paper II: Donahue and Simonyan (2019) Large Scale Adversarial Representation Learning (Big BiGAN) (BiGAN, concurrent and similar to ALI), NeurIPS2019
Lecturers: Shanel Gauthier and Abderrahim Fathan
Slides: BigBiGAN (Google Slides)
Video: here
Lecture 23 (24/11/2020)
Paper I: Cai et al. (2020) Are all negatives created equal in contrastive instance discrimination? ArXiv
Lecturers: Guillaume Lagrange and Nathaniel Simard
Slides: Contrastive Negatives (Google Slides)
Video: here
Paper II: Chen et al (2020) Intriguing Properties of Contrastive Losses. ArXiv
Lecturers: Arnaud L'Heureux and Raghav Gupta
Slides: Contrastive Properties (Google Slides)
Video: here
Lecture 24 (26/11/2020)
Paper I: Orhan et al (2020) Self-supervised learning through the eyes of a child NeurIPS 2020
Lecturers: Maxime Daigle and Zhen Liu
Slides: here
Video: here
Paper II: Lynch and Sermanet (2020) Grounding Language in Play ArXiv
Lecturer: Simon Ramstedt
Slides: LfP (Google Slides)
Video: here