Deep Learning:
Theory & Data Science

• What Is This Course About?

• Canonical datasets

• Prototypical Networks

• Optimization

• Sparsity and Information Compression

• Failure of Theory

• Adversarial Examples

• Introduction to Logistic Regression

• Max-Margin Implicit Bias of Gradient Descent on Separable Data

• Linear and Quadratic Discriminant Analysis

• Nonlinear Discriminant Analysis

• Empirical Evidence

• Optimality

• Relation to Max-Margin and Discriminant Analysis

• Unconstrained Features Model for MSE Loss

• Related Works

• Boomlet in Studies of Deep Net Spectra

• Introduction to Random Matrix Theory

• Layer-Wise Study of Features

• Layer-Wise Study of Backpropagated Errors

• Measuring Deep Net Spectra at Scale

• Study of Fisher Information Matrix and Hessian

• Introduction to Kernels

• Neural Network Gaussian Processes

• Neural Tangent Kernel

• Lazy Versus Active Training

• Max-Margin and Sparsity Implicit Bias

• Introduction to sparsity

• Sparse Coding

• Convolutional Sparse Coding

• Multilayered Convolutional Sparse Coding

Lecture 3
Lecture 4
Lecture 5
Lecture 6
Lecture 7
Lecture 8
Lecture 9
Lecture 10
Lecture 11
Lecture 12
Lecture 13
Lecture 14
Lecture 15
Lecture 16
Lecture 17
Lecture 18
Lecture 19
Lecture 20
Lecture 21
Lecture 22
Lecture 23
Lecture 24

HW1
HW2
HW3
HW4
HW5
HW6
HW7
HW8
HW9

HW1

• Preferably on the topic of theoretical or empirical investigation of deep learning.

• Examples of papers can be found below. However, you are encouraged to pick a paper not included in the list.

• You can consult me about your choice during office hours, via email, or through other communication channels.

• In the final 4 lectures of the course, students will present their projects.

• Students will have 5 minutes of presentation followed by 1 minute of questions.

• Pairs will have 10 minutes of presentation followed by 2 minutes of questions.

• The presentation should use slides (keynote, google slides, powerpoint, beamer, or other similar tools).

• The presentation should summarize the report.

• Students will submit a two-page report:

  • 1 page summarizing the paper

  • 1 page proposing and implementing a novel experiment or proving a theoretical result.

  • You are encouraged to build your experiment on open-source implementations, if available.

• Pairs will submit a twice longer report.

• PDF format, 1-inch margins, font size 10pt, preferably typed in Latex.

• The deadline is midnight on April 23rd.