One World Seminar on Mathematics of Machine Learning

Diffusion models represent a remarkable new class of deep generative models, yet the mathematical principles underlying their generalization from finite training data are poorly understood. This talk offers novel theoretical insights into diffusion model generalization through the lens of "model reproducibility," revealing a surprising phase transition from memorization to generalization during training, notably occurring without the curse of dimensionality. Our theoretical framework hinges on two crucial observations: (i) the intrinsic low dimensionality of image datasets and (ii) the emergent low-rank property of the denoising autoencoder within trained neural networks. Under simplified settings, we rigorously establish that optimizing the training loss of diffusion models is mathematically equivalent to solving a canonical subspace clustering problem. This insight quantifies the minimal sample requirements for learning low-dimensional distributions, scaling linearly with the intrinsic dimension. Furthermore, by investigating this under a nonlinear two-layer network, we fully explain the memorization-to-generalization transition, highlighting inductive biases in learning dynamics and the models' strong representation learning ability. These theoretical insights have profound practical implications, enabling various applications for generation control and safety, including concept steering, watermarking, and memorization detection. This work not only advances theoretical understanding but also stimulates numerous directions for many applications in engineering and science.

Ensemble methods combine predictions from various statistical learning models. Their most famous representatives are random forests or neural network ensembles. This talk will center around the question: "How many models should I aggregate?" We will see that the answer depends on the chosen performance metric. Specifically, in the case of convex losses (such as cross-entropy in classification or mean squared error in regression), the error is a decreasing function of the number of models. In the case of non-convex losses (such as classification error in classification or the Fréchet Inception distance in generative modelling), things are more nuanced, and the error can sometimes be non-monotonic.  These results will be illustrated with examples of neural network ensembles. This work is notably based on the paper JMLR paper "Are Ensembles Getting Better All the Time?" (http://jmlr.org/papers/v26/24-0408.html), joint work with Damien Garreau (Julius-Maximilians-Universität Würzburg). It will also feature recent work with Raphaël Razafindralambo, Rémy Sun, and Frédéric Precioso (Inria, Université Côte d'Azur).