Deep Learning:
Theory & Data Science
University of Toronto
Department of Mathematics
Fall 2023
Vardan Papyan
vardan.papyan@utoronto.ca
Course Description
Deep learning systems have revolutionized field after another, leading to unprecedented empirical performance. Yet, their intricate structure led most practitioners and researchers to regard them as blackboxes, with little that could be understood. In this course, we will review experimental and theoretical works aiming to improve our understanding of modern deep learning systems.
Evaluation
[20%]: Attendance and participation
[40%]: Tiny PyTorch coding exercises
[40%]: Final project on paper of your choosing
Lecture Slides
What Is This Course About?
Introduction to Deep Learning
Batch Normalization
Optimization
Transformers
Mysteries in Deep Learning
Information Bottleneck
Criticism of the Information Bottleneck
Rethinking Generalization
Implicit Bias of Gradient Descent on Separable Data
Neural Collapse
Neural Collapse: Unconstrained Features Model
Neural Network Gaussian Process
Neural Tangent Kernel
Predicting Generalization With NTK
Lazy Versus Active Learning
Project
Choose a paper
On the topic of theoretical or empirical investigation of deep learning.
You can consult me about your choice during office hours, via email, or through other communication channels.
Submit a two-page report
PDF format, 1-inch margins, font size 10pt, preferably typed in Latex.
1 page summarizing the paper.
1 page proving a novel theoretical result or proposing and implementing a novel experiment. You are encouraged to build your experiment on open-source implementations, if available.
Deadline is last day of the semester.
Present the report in the final lectures
Using slides (keynote, google slides, powerpoint, beamer, or other similar tools).
5 minute presentation of paper summary and your novelty followed by 1 minute of questions.
Pairs:
Twice longer report.
10 minutes of presentation followed by 2 minutes of questions.