Why Do Neural Networks Learn Simple Things First?

Imagine you are learning to draw a portrait. You would not start by perfecting every fine hair and wrinkle — you would begin with a rough outline, getting the overall shape right, and then gradually add finer detail. It turns out that deep neural networks (DNNs) behave in a strikingly similar way when they learn. This behavior, discovered by Xu et al. and independently confirmed by Rahaman et al., is called the Frequency Principle (or F-Principle).

In plain terms: during training, neural networks learn low-frequency (smooth, broad) patterns in the data first, and only later begin to capture high-frequency (sharp, fine-grained) patterns. This paper, published in Communications in Computational Physics, demonstrates this principle systematically and explains why it happens.

Before going further, it is important to clear up a potential confusion. When we talk about “frequency” in the context of images, most people think of how quickly pixel brightness changes across the image — smooth skies are low-frequency, sharp edges are high-frequency. But that is NOT what this paper means.

Instead, the paper focuses on response frequency: how rapidly the network’s output changes in response to changes in the input. Think of it this way:

•         A low-frequency response means: small changes in the input produce small, gradual changes in the output. The model has learned a smooth, general pattern.

•         A high-frequency response means: small changes in the input produce sharp, abrupt changes in the output. The model has captured fine-grained detail.

A simple illustration: if a DNN is trained to fit the function y = sin(x) + sin(3x) + sin(5x), the F-Principle tells us the network will first learn the slow wave sin(x), then gradually pick up sin(3x), and only much later fully capture sin(5x). The researchers verified this precisely, tracking the error at each frequency component over thousands of training steps.

What makes this finding significant is its universality. The researchers tested the F-Principle across a wide range of conditions:

•         Different datasets: MNIST (handwritten digits) and CIFAR10 (small color images) — two of the most commonly used benchmarks in machine learning.

•         Different architectures: Fully connected networks, convolutional neural networks (CNNs), and even the large VGG16 network.

•         Different activation functions: Both tanh and ReLU activations.

•         Different loss functions: Mean squared error, cross-entropy, and variational losses.

In every single case, the same pattern emerged: the low-frequency components of the target function were learned early, while high-frequency components lagged behind. This was not a quirk of one setting — it is a fundamental property of how gradient-based training interacts with the architecture of neural networks.

Measuring frequency behavior in high-dimensional data (like 784-pixel MNIST images) is genuinely hard. The standard Fourier transform becomes computationally impossible at that scale. The authors developed two clever workarounds:

1. The Projection Method

Instead of computing the full high-dimensional Fourier transform, the researchers “projected” the data onto a single direction (the first principal component of the input space) and computed a simpler one-dimensional Fourier transform along that direction. This allowed them to track how specific frequency components converged during training without prohibitive computation.

2. The Filtering Method

The filtering approach is more elegant. The idea: blur the training labels with a Gaussian filter to create a “smoothed” (low-frequency) version of the data, and subtract that to get the “sharp” (high-frequency) residual. Then, during training, separately track how fast the network learns each part. Consistently, the smooth part was learned much faster than the sharp part — confirming the F-Principle at a coarse-grained level across all directions simultaneously.

The paper offers a theoretical explanation, at least for a simplified setting. The key insight is this: activation functions like tanh are smooth and well-behaved. Their Fourier transforms decay rapidly at high frequencies. In other words, a neural network built from smooth activations naturally has a bias toward representing smooth (low-frequency) functions.

More formally, the authors prove (Theorems 1 and 2) that for a one-hidden-layer network with tanh activations and small initial weights, the gradient of the loss with respect to network parameters is exponentially larger for lower frequencies than higher ones. This means gradient descent will reduce low-frequency errors much faster — the math essentially forces the network to learn simple patterns before complex ones.

Put intuitively: the network’s internal machinery finds it “easier” to adjust itself to capture broad strokes. Fine detail requires the parameters to take on very specific values, and gradient descent takes longer to navigate toward those configurations.

The paper highlights a striking contrast between neural networks and a classical numerical method called the Jacobi iteration, commonly used to solve systems of equations in scientific computing. The Jacobi method does the opposite of the F-Principle — it converges on high frequencies first and struggles with low-frequency errors.

This difference is not just a theoretical curiosity. The authors propose a hybrid approach: use a neural network first to quickly learn the low-frequency structure of a problem, then hand off to Jacobi (which is fast at high frequencies) to clean up the fine detail. In their experiments solving Poisson’s equation — a classical physics problem — this combination converged significantly faster than either method alone. This is a concrete example of how understanding a model’s bias can lead directly to practical algorithmic improvements.

This paper is worth understanding deeply for several reasons:

•         Neural networks have an implicit bias. They do not just “learn” in a neutral way — they are systematically biased toward low-frequency functions. Understanding this bias helps you anticipate when a model will and will not work well.

•         Fourier analysis is a powerful lens. Much of what makes neural networks tick can be understood through the frequency content of the data and the model’s output. This framing complements the usual loss-curve perspective.

•         Knowing a model’s bias enables better design. The hybrid DNN + Jacobi solver is a perfect example: once you know the DNN is great at low frequencies and weak at high ones, you can pair it with something that complements its weaknesses.

•         Generalization is not magic. Neural networks generalize well on typical datasets not because they are mysteriously powerful, but because those datasets happen to match the network’s bias. When the data does not match the bias (high-frequency problems), generalization falls apart.

The Frequency Principle is one of those ideas that, once you know it, changes how you think about deep learning entirely. It reframes training not as a black-box optimization, but as a structured, predictable process with a clear direction: broad strokes first, fine detail later. That predictability is intellectually satisfying, but more importantly, it is useful. It helps researchers design better algorithms, predict failure modes, and build hybrid methods that play to the strengths of different tools.

If you want to go deeper, the original paper by Xu et al. is remarkably readable for a theoretical deep learning paper, and the subsequent work by Luo et al. (2019) extends the theory to general DNN architectures. Both are well worth your time.