One World Seminar on Mathematics of Machine Learning

Abstract: Diffusion models now underpin many of the most powerful generative systems, yet understanding the mechanisms that prevent their memorization of training data and allow generalization remains a key challenge. In this talk, I will focus on the role of the training dynamics in the transition from generalization to memorization. I will identify two sharply separated timescales in training. The first, τ_gen, marks the onset of high-quality sample generation and is largely insensitive to dataset size. The second, τ_mem, signals the emergence of memorization and grows linearly with the number of training examples. As a result, increasing the dataset size creates an expanding window of training times during which models generalize effectively—even though the same models will exhibit strong memorization if optimized for longer. Only when the dataset exceeds a model-dependent threshold does memorisation vanish altogether in the infinite-time limit. These findings reveal a form of implicit dynamical regularization: the training dynamics themselves protect against memorization over a substantial and controllable regime, despite extreme overparameterization. I will support these conclusions with numerical experiments using standard U-Net architectures on both synthetic and realistic datasets, and with a complementary theoretical analysis of a tractable random-features model in the high-dimensional limit. Together, the results offer a unifying framework for understanding how dataset size and training time jointly govern generalization in modern diffusion models. This talk is based on a joint work presented at NeurIPS 2025 with T. Bonnaire, R. Urfin and M. Mézard.

Abstract: The search for the optimal shape parameter for Radial Basis Function (RBF) kernel methods has been an outstanding research problem for decades. In this work, we establish a theoretical framework for this problem by leveraging a recently established theory on sharp direct, inverse and saturation statements for kernel based approximation. In particular, we link the search for the optimal shape parameter to superconvergence phenomena.

The analysis is carried out for finitely smooth Sobolev kernels, thereby covering large classes of radial kernels used in practice, including those emerging from current machine-learning methodologies.The results elucidate how approximation regimes, kernel regularity, and parameter choices interact, thereby clarifying a question that has remained unresolved for decades.