CSCI 699: Dynamics of Representation Learning

No person steps in the same river twice, for it’s not the same river and not the same person.

-Heraclitus (500 BC, after being asked whether the test set will resemble the training set)

Deep learning is a swiftly changing field, and many aspects of DL can be viewed as dynamical processes. Optimizers used in deep learning, for instance, exploit dynamics that lead to solutions with desirable properties that are not intrinsic to the loss function itself. Sampling relies on MCMC dynamics and finds applications in Bayesian optimization, latent factor inference, and energy-based probability modeling. Effective learning must be dynamic to reflect a world that is constantly changing. Deep architectures and normalizing flows can be viewed as dynamical processes that transform data over time through a sequence of layers. This course will be research-oriented and survey recent papers in the field. Rather than focusing on state-of-the-art numbers on popular benchmarks, we will look for common mathematical threads and try to develop deeper intuition about representation learning through the lens of dynamics.

Syllabus

Some pedagogical background on topics in the syllabus can be found in the Deep Learning Book (DLB) or Dive into Deep Learning (DIDL). Some related material comes from Roger Grosse's excellent (and more optimization focused) class Neural Net Training Dynamics (NNTD). Each lecture has some associated readings in the syllabus below.

Update May 17, 2022: The class is complete! Many thanks to the amazing guest speakers, and to the students for some fascinating and innovative class projects. Please contact me if you are interested in accessing particular lectures (I can't share guest lectures because I didn't ask for permission to do that.)