Outline

• Introduction

• Historical Context of Representation Learning

• Speech Representation Learning Paradigms

  • Generative Approaches

  • Contrastive Approaches

  • Predictive Approaches

• Learning from Multimodal Data

• Benchmarks for Self-Supervised Learning Approaches

• Analysis of Self-Supervised Representations

• From Representation Learning to Zero Resources

  • Unsupervised Speech Recognition

  • ASR-TTS Technique

  • Zero Resource Speech Technologies and Challenges

  • Textless NLP

• Beyond Accuracy

  • How to Use SSL Models

  • Security Issues

  • Data Bias

  • Compressing SSL Model

• Toolkits for speech SSL

• Concluding Remarks

Reading list

• General Introduction to Self-supervised Learning

  • Linus Ericsson, Henry Gouk, Chen Change Loy, Timothy M. Hospedales, Self-Supervised Representation Learning: Introduction, Advances and Challenges, arXiv:2110.09327

• Self-supervised Learning for Natural Language Processing

  • Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, Graham Neubig, Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing, arXiv:2107.13586

  • Anna Rogers, Olga Kovaleva, Anna Rumshisky, A Primer in BERTology: What we know about how BERT works, TACL, 2020

  • Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, Xuanjing Huang, Pre-trained Models for Natural Language Processing: A Survey, arXiv:2003.08271

• Self-supervised Learning for Speech Processing

  • Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D. Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, Tara N. Sainath, Shinji Watanabe, Self-Supervised Speech Representation Learning: A Review, arxiv.2205.10643