General Frequency Principle of Gradient Descent: High-Frequency Suppression and Low-Frequency Priority under Linear Approximation

Introduction: Extending the Frequency Principle

In machine learning theory, the implicit bias of gradient descent training has long been a research hotspot. A recent paper, General Frequency Principle for Linear Approximation Classes, generalizes the frequency principle from neural networks to finite linear approximation classes, revealing how gradient descent treats high- and low-frequency components differently during training. The study validates its conclusions through theoretical proofs and numerical experiments (using Legendre and trigonometric bases), offering a new perspective for understanding the learning dynamics of linear models.

Core Theoretical Results

Amplitude Decay of High-Frequency Components

The paper proves that, under appropriate assumptions, the amplitude of the learned solution at high frequencies tends to zero consistently throughout training relative to the target amplitude. This means gradient descent tends to fit low-frequency components first, while the amplitude of high-frequency components is always suppressed. This conclusion does not depend on training time and is global in nature.

Frequency Principle is an empirical observation in deep learning theory: during gradient descent training, neural networks tend to fit the low-frequency components of the target function (global, smooth structures) first, then gradually fit the high-frequency components (local, sharp variations), and the amplitude of high-frequency components is often suppressed. This principle was first systematically observed in deep fully connected networks and convolutional networks, and is known as the "F-Principle." Common explanations include: spectral bias of gradient descent—the Hessian matrix of the loss function has larger eigenvalues in low-frequency directions, making low-frequency directions converge faster; and implicit regularization from parameter initialization. The paper generalizes it to linear approximation classes, implying that the frequency principle is not unique to neural networks but an intrinsic property of gradient descent in a broader function space.

Sufficient Conditions for Low-Frequency Priority

The authors establish sufficient conditions for low-frequency priority, covering two scenarios: training a single-frequency target alone, and training two frequency components in the same run. These conditions ensure that low-frequency components are learned first.

Interestingly, this general frequency principle does not imply that "low frequencies always converge faster." Instead, for two frequencies satisfying a specific comparison condition, one component requires a longer learning time, and in common cases, the component with longer learning time corresponds to a higher frequency. This distinction clarifies a common misunderstanding of the frequency principle.

Numerical Experimental Validation

The paper conducts numerical experiments using Legendre and trigonometric bases, validating the theoretical results. Experiments show that at high frequencies, the learned amplitude indeed tends to zero relative to the target amplitude, and low-frequency components are fitted first. These experiments are consistent with the theoretical analysis, enhancing the credibility of the conclusions.

Legendre basis is a set of orthogonal polynomials defined on the interval [-1, 1], generated by the recurrence relation P_{n+1}(x) = [(2n+1)x P_n(x) - n P_{n-1}(x)]/(n+1), with the first few terms being P_0=1, P_1=x, P_2=(3x^2-1)/2. It has the orthogonality property ∫{-1}^{1} P_m(x)P_n(x) dx = 2/(2n+1) δ{mn}, and P_n is exactly a polynomial of degree n, so {P_n} forms a basis of the polynomial space. In numerical analysis, the Legendre basis is often used for function approximation and spectral methods because its orthogonality makes projection coefficient computation stable, and the decay rate of coefficients reflects the smoothness of the target. The paper chooses the Legendre basis as an instance of linear approximation classes to examine the learning order of different "frequency" (corresponding to polynomial degree) components under gradient descent.

Trigonometric basis, i.e., the Fourier basis {1, cos(nπx), sin(nπx)} or complex exponentials {e^{inπx}}, is an orthogonal basis of the periodic function space. Its coefficients are Fourier coefficients, and frequency n directly corresponds to the oscillation rate, making it the most natural basis for studying the frequency principle. The two bases represent non-periodic (polynomial) and periodic (trigonometric) approximation scenarios, respectively, complementarily validating the universality of the theoretical conclusions.

Implications for Machine Learning Practice

This general frequency principle has significance for understanding the behavior of linear models and kernel methods. It shows that gradient descent, when training linear approximators, naturally produces a regularization effect that suppresses high-frequency noise and prioritizes learning low-frequency structures. This helps explain why linear models can still generalize in overparameterized settings. At the same time, the principle reminds practitioners that in tasks requiring high-frequency details, optimization strategies may need adjustment or adaptive methods may be necessary.

Conclusion

The paper generalizes the frequency principle to finite linear approximation classes, providing theoretical guarantees and experimental validation. Its conclusions emphasize the suppression of high-frequency components by gradient descent and the conditionality of low-frequency priority, offering a more general foundation for research on the frequency principle in deep learning theory. Future work can explore the application of this principle in more complex models.

One-line Summary

The paper generalizes the frequency principle to finite linear approximation classes, proving that gradient descent suppresses high-frequency amplitudes and establishing sufficient conditions for low-frequency priority.

Paragraph Summary

This paper studies the implicit bias of gradient descent when training finite linear approximation classes, generalizing the frequency principle known in neural networks to general linear models. The core conclusion is: under appropriate assumptions, the amplitude of the learned solution at high frequencies consistently tends to zero relative to the target amplitude throughout training, i.e., gradient descent suppresses high-frequency components; the authors further establish sufficient conditions for low-frequency priority, covering both single-frequency targets and simultaneous training of two frequencies. The paper clarifies a common misunderstanding—the frequency principle does not imply that low frequencies always converge faster: for two frequencies satisfying specific comparison conditions, in common cases the component with longer learning time corresponds to a higher frequency. Numerical experiments using Legendre and trigonometric bases validate the theoretical results, providing a new perspective for understanding the generalization and regularization effects of linear models and kernel methods.