Universality of Gradient Descent Training: Does Neural Network Architecture Choice Really Matter?

Introduction: A Neglected Theoretical Question

In deep learning practice, we invest enormous effort designing complex neural network architectures—from convolutional networks to Transformers, from residual connections to attention mechanisms. Yet a fundamental theoretical question is often overlooked: To what extent does architecture choice determine what a model can learn? If the training algorithm itself (such as gradient descent) possesses a certain "universality," should we reconsider our obsession with architecture design?

A recent study titled Universality of Gradient Descent Neural Network Training attempts to answer this question from a theoretical perspective. Its core proposition—the universal approximation capability of gradient descent training—touches the theoretical foundations of deep learning.

What Is "Universality of Training"?

Starting from the Universal Approximation Theorem

To understand this research, we need to first revisit the classic Universal Approximation Theorem. This theorem tells us that a single hidden-layer feedforward neural network with sufficient neurons can approximate any continuous function to arbitrary precision.

The Universal Approximation Theorem was first proven by George Cybenko in 1989 for sigmoid activation functions, and later generalized by Kurt Hornik et al. in 1991 to more general activation functions. The core idea draws from the Stone-Weierstrass theorem and Kolmogorov superposition theorem in functional analysis. Notably, this theorem is a purely existential result—it guarantees the existence of an approximation scheme but provides no tight upper bound on the number of neurons required. Subsequent research has shown that for certain function classes, the required number of neurons may grow exponentially with input dimension, the so-called "curse of dimensionality." The emergence of deep networks partially alleviates this problem: work by Telgarsky and others proved that deep networks have exponential parameter efficiency advantages over shallow networks of equal width for representing certain functions, providing another theoretical angle for explaining the practical success of deep learning.

However, this classic theorem has a critical limitation: it only proves the "existence" of a set of weights that can achieve approximation, but does not indicate whether these weights can be found through practical training algorithms (such as gradient descent). In other words, there is a gap between theoretical expressiveness and practical learnability.

The Core Claim of Training Universality

The concept of "universality of gradient descent training" attempts to bridge this gap. It no longer asks "can the network represent a certain function," but rather "through gradient descent optimization, can the network actually converge to a solution that approximates the target function?"

This is a stronger and more practice-relevant proposition. In real scenarios, we never manually set weights but rely entirely on optimization algorithms. If we can prove that gradient descent achieves effective approximation across a wide range of architectures and initialization conditions, this would provide more solid theoretical support for the success of deep learning.

Why the Universality of Gradient Descent Matters So Much

Explaining Deep Learning's "Unexpected Success"

The practical success of deep learning has always been accompanied by theoretical puzzlement. Neural network loss functions are typically highly non-convex, and according to traditional optimization theory, gradient descent should easily get trapped in poor local optima. Yet in reality, even randomly initialized large networks consistently find solutions that generalize well under gradient descent.

Traditional optimization theory takes a pessimistic stance on non-convex problems, since finding the global optimum of a non-convex function is NP-hard in general. The loss surface of neural networks is particularly complex: a network with n parameters has its loss function defined on an n-dimensional space, potentially containing an exponential number of saddle points and local minima. However, Choromanska et al. (2015), borrowing tools from random matrix theory in statistical physics, found that in over-parameterized networks, most local minima have loss values very close to the global optimum, while high-loss local minima are extremely rare. Furthermore, Dauphin et al. pointed out that in high-dimensional spaces, saddle points are far more prevalent than local minima, and gradient descent variants (such as SGD with momentum, Adam, and other adaptive methods) can effectively escape saddle points. These findings collectively form the theoretical puzzle explaining why gradient descent "unexpectedly succeeds" on non-convex loss surfaces.

Research on training universality argues mathematically that under certain conditions (such as sufficiently wide networks and reasonable initialization), this "good fortune" of gradient descent is not coincidental but a theoretically inevitable universal phenomenon. This aligns with the much-discussed Neural Tangent Kernel (NTK) theory—in the infinite-width limit, network training dynamics become linearized and convergence behavior becomes predictable.

The Neural Tangent Kernel theory, proposed by Arthur Jacot et al. in 2018, is a key framework for understanding wide network training dynamics. Its core insight is that as network width approaches infinity, changes in network output with respect to parameters can be precisely described by a first-order Taylor expansion, and the training process degenerates into a kernel regression problem. Specifically, defining the neural tangent kernel K(x,x') = ⟨∇_θf(x,θ), ∇_θf(x',θ)⟩, in the infinite-width limit this kernel remains constant during training (called the "lazy training" regime), making gradient descent dynamics a linear ODE whose convergence can be guaranteed by the minimum eigenvalue of the kernel matrix. However, NTK theory also faces criticism: actual finite-width networks exhibit "feature learning" capability, where intermediate layer representations change significantly during training—precisely what the NTK framework cannot capture. Greg Yang et al.'s μP (Maximal Update Parameterization) theory and mean-field theory attempt to achieve theoretical analysis while preserving feature learning, representing frontier directions beyond NTK.

Implications for Neural Network Architecture Design

If training universality holds, an intriguing corollary is that different architectures may exhibit a certain equivalence in their ultimate approximation capability. This doesn't mean architecture is unimportant—architecture determines training efficiency, sample complexity, and inductive bias—but it suggests that some seemingly critical design choices may have no essential difference in expressiveness under sufficient training.

Inductive bias refers to a learning algorithm's prior preference for certain solutions within the hypothesis space, and it plays a central role in architecture selection. Different architectures encode different inductive biases: Convolutional Neural Networks (CNNs) encode translation invariance and locality through local connectivity and weight sharing, making them naturally suited for image tasks; Recurrent Neural Networks (RNNs) encode the temporal structure of sequences; Transformer's self-attention mechanism encodes global interaction capability among set elements; Graph Neural Networks (GNNs) encode permutation equivariance of graphs. These inductive biases determine a model's generalization ability and sample efficiency under limited data. Even if training universality guarantees that different architectures can ultimately approximate the target function, under practical constraints of data volume and computational budget, matching the inductive bias to the problem structure can mean orders-of-magnitude efficiency differences. This is also why Vision Transformer (ViT) surpasses CNNs on large-scale datasets but underperforms on small datasets—lacking image-specific inductive biases, ViT needs more data to "learn" the prior knowledge that CNNs inherently possess.

This perspective helps us view the constant stream of architectural innovations more rationally: the real value often lies in optimization efficiency and matching of inductive biases, rather than in the upper bound of expressiveness.

The Gap Between Theory and Practice: Limitations of Universality Proofs

Common Prerequisites of Universality Proofs

We need to clearly recognize that such universality results are typically built on a series of idealized assumptions:

• Infinite or extremely large width: Many proofs rely on network width approaching infinity, while practical networks have finite scale.
• Specific initialization schemes: Conclusions are often sensitive to the distribution of weight initialization.
• Smoothness and continuity assumptions: Target functions must satisfy certain regularity conditions.
• Idealized optimization processes: Ignoring stochastic gradient noise, finite learning rates, and other practical factors.

Among these, the over-parameterization condition particularly deserves deeper discussion. Over-parameterization means the number of network parameters far exceeds the number of training samples. Traditional statistical learning theory (such as VC dimension theory, Rademacher complexity) predicts that over-parameterized models should severely overfit, but deep learning practice shows the opposite. This "double descent" phenomenon has sparked extensive theoretical discussion. A key explanatory direction is "implicit regularization": the gradient descent algorithm itself prefers certain types of solutions. For example, for linear models, gradient descent from zero initialization converges to the minimum-norm solution; for matrix factorization problems, gradient descent tends to find low-rank solutions. In neural networks, the specific form of implicit regularization is more complex, potentially involving function complexity measures (such as path norm, Fisher-Rao norm, etc.), but the core idea is that the optimization algorithm not only finds solutions that fit the training data but also "automatically" selects solutions with good generalization properties.

These prerequisites mean that there remains a significant gap between theoretical universality and engineering practice. A conclusion that holds in the infinite-width limit may not directly guide us in training a billion-parameter language model.

The True Value of Foundational Theoretical Research

Nevertheless, the significance of such foundational theoretical research should not be underestimated. What they provide is a framework for understanding: helping us clarify "why deep learning works," delineating boundaries between possible and impossible, and providing first-principles guidance for algorithm design. When we encounter problems like training non-convergence or poor generalization in practice, solid theory often provides deeper diagnostic perspectives.

It's worth noting that the value of theoretical research often manifests in engineering practice in nonlinear ways. For example, the residual connections proposed by He et al. were partially inspired by theoretical analysis of vanishing/exploding gradients; the success of Batch Normalization, initially attributed to "internal covariate shift," was later revealed by theoretical work to likely operate through smoothing the loss surface; and the learning rate warmup strategies and parameterization schemes widely adopted in recent large language model training can similarly be traced back to mathematical analysis of training dynamics.

Conclusion: From Empirical Alchemy to Rational Design

In today's increasingly fierce large model arms race, the pace of engineering iteration far outstrips theoretical understanding. We're accustomed to tuning hyperparameters and scaling up data in an empirical fashion, rarely pausing to ask about the underlying mathematical principles.

Research on gradient descent training universality reminds us that the success of deep learning should not be viewed as pure black-box magic. Why gradient descent works, where the boundaries of architecture choice lie, how optimization and expressiveness interact—the answers to these fundamental questions will ultimately determine whether we can move from "trial-and-error engineering" to "rational design."

For researchers and practitioners alike, paying attention to these foundational theoretical advances may hold greater long-term value than chasing the next hot architecture. Just as theoretical physics and experimental physics mutually reinforce each other, the field of deep learning equally needs deep integration of theoretical insight and engineering intuition. Only when we truly understand why gradient descent can "find the right path" on high-dimensional non-convex surfaces will we be able to design more efficient, more reliable, and more interpretable learning systems.