The Geometry of Random Neural Networks
In the infinite-width limit, deep neural networks induce isotropic Gaussian fields on the sphere, whose covariance encodes key information about architecture and activation function. This talk presents a unified framework, based on three recent works, showing a robust trichotomy — sparse, low-disorder, high-disorder — across spectral and geometric descriptors of these fields. First, we classify activation functions via the angular power spectrum of the limiting field. In the low-disorder phase, the field converges to a constant, both in L2 and in Sobolev norms; in the sparse phase, L2 convergence persists but Sobolev norms diverge, signaling increasingly sharp high-frequency components despite global L2 stability; in the high-disorder phase, the spectrum drifts to arbitrarily high multipoles, with no Sobolev control at any fixed order. Second, we study level-set boundaries: non-smooth activations yield fractal boundaries with depth-increasing Hausdorff dimension, while smoother ones show contraction, stability, or exponential growth, mirroring the spectral regimes. Third, using Kac–Rice theory, we derive asymptotics for the expected number of critical points, again recovering the same trichotomy. These results reveal a universal mechanism governed by the covariance kernel's local behavior, supported throughout by numerical simulations.
Joint work with Domenico Marinucci, Michele Salvi, and Stefano Vigogna.