Dynamical Systems in Machine Learning (DSML)

Arthur Pellegrino (ENS-PSL, PhD in Informatics), Alex Cayco-Gajic (ENS-PSL, PhD in Applied Mathematics)

Announcements

05/10/26 - The slides from the IASD course presentation day can be found: here

Overview

From system identification and time-series analysis to the study of optimisation dynamics in neural networks, dynamical systems have become increasingly central to modern machine learning. Such rapid advances at the intersection of dynamical systems theory and machine learning are creating exciting new opportunities for applied mathematicians and computer scientists alike. In DSML, students will explore how ordinary and stochastic differential equations can be formulated, analysed, and applied to machine learning, developing a deeper understanding of the mathematical principles underlying modern learning algorithms.

Inference of dynamical systems from scientific data

Recent advances in machine learning have made it possible to infer dynamical systems governed by differential equations directly from time-series data. Methods such as NeuralODEs and SINDy can be used to learn or identify the underlying governing equations from sparse and noisy observations of the state of a system. In DSML, we will cover topics including differentiable ODE and SDE solvers, which enable gradients to be propagated through numerical integration; variational methods for dynamical-system inference; and Koopman theory, which represents nonlinear dynamics through linear operators acting in an infinite-dimensional function space. We will motivate this with applications to physics, biological and chemistry data.

Modern diffusion models, such as Stable Diffusion, can be formulated in terms of stochastic differential equations: a forward SDE that progressively transforms data into noise, and a learned reverse-time SDE that transforms noise into generated samples. Rather than focusing on the derivation of the training objectives for diffusion and flow-matching models, which is covered in other courses, we will instead use dynamical systems tools to reverse-engineer trained generative models. For example, we will study how diffusion models learn low-dimensional data manifolds, and how the generation trajectory can be divided into distinct dynamical regimes associated with generating novel samples versus reproducing training data.

Mathematical foundations of generative models

Deep neural network learning theory

Learning theory provides mathematical tools for understanding the dynamics of neural network optimisation and the implicit biases induced by training algorithms. We will show how gradient descent can be viewed, in the continuous-time limit, as an autonomous dynamical system. By studying the resulting gradient flow and characterising its fixed points and stability, dynamical systems theory provides a framework for understanding neural network solutions as attractors of the optimisation dynamics. We will also examine how features of loss dynamics, such as sequential learning, can be understood as transitions between saddle points and other critical points of the loss landscape.

DSML will demonstrate the practical relevance of these concepts through successful applications in “AI4Science”, including neuroscience, genomics, protein folding, and other areas of biology and physics, as well as in generative modelling of images and video. By the end of the course, students will have developed the theoretical skills necessary to engage with the dynamical systems tools that underpin modern machine learning and artificial intelligence.

Tentative course schedule

Week 1: Overview of dynamical systems theory

Topics covered:

Vector fields & flows
Invariant manifolds
Fixed points & Stability

Weeks 2-4: Dynamical system inference from data

Topics covered:

Topological conjugacy and Taken's theorem
Koopman operators
Sparse identification of nonlinear dynamics & Neural ODEs
Differentiable ODE/SDE solvers
Backpropagation-through-time & Adjoint methods
Variational inference and filtering of linear dynamical systems

Weeks 5-6: Generative models as SDEs

Topics covered:

Liouville & Fokker-Plank equations
Linear time-varying dynamical systems
Attractor manifolds & inference of data manifolds
Dynamical phase transitions in diffusion dynamics

Weeks 7-8: Learning dynamics

Topics covered:

Approximation of SGD as an SDE
Saddle-to-saddle dynamics & Sequential learning
Reduced-rank dynamics & Summary statistics
Non-stationary SGD & Symmetries

Learning outcomes

Analyse dynamical properties of models, including stability, identifiability, and long-term behaviour.
Formulate generative models as dynamical systems using ODEs and SDEs.
Infer latent dynamics from real-world high-dimensional time-series data.
Mathematically analyse deep neural networks training dynamics in the continuous-time limits.

Pre-requisites

Linear algebra, multivariate calculus, advanced probability.

Chapter 1 of this textbook offers a good overview of the background math we will need.

Course modalities

The course consists of eight 3h sessions. In weeks 4, 6, and 8, the session includes a shorter 2h lecture followed by a related 30min quiz (20% each). In other weeks we will have pen-and-paper exercise sheets (2% each, graded only on participation). The course concludes with a final research project (30%, oral presentation).

Readings

Review

Meiss, J. D. (2007). Differential dynamical systems. Society for Industrial and Applied Mathematics (SIAM).

Part I.

Chen, R. T., Rubanova, Y., Bettencourt, J., & Duvenaud, D. K. (2018). Neural ordinary differential equations. Advances in neural information processing systems (NeurIPS).
Gu, A., Goel, K., & Re, C. (2022). Efficiently Modeling Long Sequences with Structured State Spaces. International Conference on Learning Representations (ICLR).
Brunton, S. L., Proctor, J. L., & Kutz, J. N. (2016). Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences (PNAS).

Part II.

Duncker, L., Bohner, G., Boussard, J., & Sahani, M. (2019). Learning interpretable continuous-time models of latent stochastic dynamical systems. International conference on machine learning (ICML).
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020). Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations (ICLR).
Rasul, K., Seward, C., Schuster, I., & Vollgraf, R. (2021). Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. International conference on machine learning (ICML).

Part III.

Saxe, A. M., McClelland, J. L., & Ganguli, S. (2013). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. International Conference on Learning Representations (ICLR).
Jacot, A., Ged, F., Şimşek, B., Hongler, C., & Gabriel, F. (2021). Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity. International conference on machine learning (ICML).
Pashakhanloo, F., & Koulakov, A. (2023). Stochastic gradient descent-induced drift of representation in a two-layer neural network. International conference on machine learning (ICML).

Page updated

Google Sites

Report abuse