12. October 2026 @ 5pm CEST / 11am EST / 8am PST [timezone converter]
Cracking the Hessian: Closed-Form Hessian Spectra for some Fundamental Neural Networks
Sidak Pal Singh, Google DeepMind, USA

Abstract: The Hessian and its spectrum hold significant theoretical and practical relevance for building op- timizers, measuring generalization, compressing models, and more. Prior works have characterized the Hessian through its spectral density, rank, and the outlier–bulk structure of its spectrum, often relying on approximations. However, the precise behavior of Hessian eigenvalues and eigenvectors remains unclear, owing both to the absence of closed-form results for non-trivial neural networks and the computational expense of empirical estimation. As a first in the literature, in this work, we derive closed-form expressions for all Hessian eigenvalues and eigenvectors in two-layer linear and ReLU networks with scalar input, arbitrary hidden width, and where the loss is aggregated over any number of samples. We further provide closed-form eigenvalues for the core component of Transformer architectures — a single self-attention layer with arbitrary sequence length. Our results reveal a previously undiscovered "paired" structure of outlier eigenvalues, a cell-wise decomposition of the Hessian spectrum with ReLU, and the sensitivity of the Hessian condition number to the query and key matrix norms, as well as the presence of attention sinks. We complement these findings with experiments beyond the assumed model setting, showing strong correlation between the largest eigenvalue and the spectral norm of weight matrices, and empirical evidence that the paired eigenvalue structure persists more generally. Overall, this advances our understanding of the Hessian via neatly exhibiting its exact and complete spectrum for a core set of networks.

OpenReview: https://openreview.net/pdf?id=gW30Rx4eZP 

26. October 2026 @ 5pm CET / 12pm EST / 9am PST [timezone converter]
Exploring and Exploiting Stability in Latent Flow Matching
Rania Briq, Jülich Supercomputing Centre and Technical University Dortmund, Germany

Abstract: In this work, we show that Latent Flow-Matching (LFM) models are robust to different types of perturbations, including data reduction and model capacity shrinkage. We characterize this stability by these models' tendency to generate similar outputs under identical noise seeds. We provide a perspective relating this phenomenon to flow matching theory, which indicates that this stability is inherent to the FM objective. We further exploit this stability to derive practical algorithms for more efficient training and inference. Concretely, first, we show that by training LFM models on significantly reduced datasets, performance is preserved, and in compute-constrained regimes, the model converges faster while maintaining quality. This yields multiple advantages, including savings in the training time due to faster convergence, and alleviating annotation effort when training conditional models. Second, LFM stability under architectural shrinkage gives rise to a two-model coarse-to-fine approach, one using a light-weight architecture for the first phase of the FM trajectory, and one with higher capacity for the second, thereby reducing the inference cost substantially. To determine which samples are informative, we introduce three sample-scoring criteria and evaluate them under standard metrics for generative models. Our results are thoroughly evaluated on multiple datasets, demonstrating the practical advantage of this stability, including data savings and a more than two-fold inference speedup while generating comparable outputs.

arXiv: https://arxiv.org/abs/2605.08398