The Theory of Deep Learning: Why Do Neural Networks Work? A Complete Overview of Theoretical Research

Introduction: The Theoretical Gap in Deep Learning

Over the past decade or so, deep learning has achieved remarkable practical successes—from image recognition to natural language processing, and now to large language models. Neural networks have reshaped nearly the entire AI field. Yet a question that has long troubled researchers persists: Do we truly understand, from a theoretical standpoint, why deep learning works?

"Theories of Deep Learning" is precisely the research field centered on this core question. Although engineering practice has raced ahead, theoretical explanations have consistently struggled to keep pace. This article surveys the key questions and major schools of thought in deep learning theory research, helping readers understand this predicament of "knowing that it works but not why."

The Core Paradox: Why Do Over-Parameterized Networks Generalize?

Classical statistical learning theory tells us that the more parameters a model has, the more prone it is to overfitting. Yet modern deep networks often possess billions or even trillions of parameters—far exceeding the number of training samples—and should, in principle, severely overfit. In reality, however, they exhibit astonishing generalization ability.

To appreciate the impact of this paradox, we first need to understand the core framework of classical statistical learning theory. Tools like VC dimension theory (proposed by Vapnik and Chervonenkis in 1971) and Rademacher complexity are built on the principle of "Occam's razor": the larger a model's capacity, the looser its generalization bound, and the more prone it is to overfitting in theory. VC dimension characterizes the maximum number of samples a class of hypothesis functions can "shatter," and is a central tool for measuring model complexity. When the number of model parameters far exceeds the number of samples, classical theory predicts that generalization error will rise sharply. Yet the actual performance of modern deep networks utterly shatters this prediction, prompting researchers to realize that traditional theoretical tools may be fundamentally unsuited to describing over-parameterized neural networks.

The Mystery of Over-Parameterization

This phenomenon is known as the "over-parameterization" paradox. Researchers have found that over-parameterized networks are actually easier to optimize and more likely to find solutions that generalize well. Some theories draw on the "double descent" curve to explain this: once model complexity crosses a certain critical point, test error decreases again, breaking with the traditional understanding of the bias-variance tradeoff.

The double descent phenomenon was systematically proposed and named by researchers including Mikhail Belkin around 2019. The bias-variance tradeoff in traditional statistical learning describes a U-shaped test error curve: a model that is too simple underfits (high bias), while one that is too complex overfits (high variance). The double descent curve, however, reveals a second descent after this U-shaped curve—once the number of model parameters exceeds the "interpolation threshold" (the critical point at which the model can perfectly fit the training data), test error actually decreases again. This phenomenon has been observed in neural networks, decision trees, and random feature models, thoroughly undermining the traditional intuition that "more complexity means more danger" and prompting researchers to reexamine the foundational assumptions of generalization theory.

Implicit Regularization: The Intrinsic Preferences of Optimization Algorithms

Another key concept is implicit regularization. Optimization algorithms such as gradient descent seem to inherently favor "simple" solutions, even without an explicit regularization term. This natural preference within the optimization process is considered one of the important sources of deep neural networks' generalization ability.

Understanding implicit regularization requires some familiarity with the different variants of gradient descent. Standard gradient descent (GD) uses the entire training dataset for each update, which is computationally expensive; stochastic gradient descent (SGD) estimates the gradient using only one or a small number of randomly sampled examples each time, introducing noise but greatly reducing computation; mini-batch gradient descent (Mini-batch GD) is a compromise between the two. Research shows that the random noise introduced by SGD is itself a source of implicit regularization—it biases the optimization trajectory toward "wide," flat minima rather than sharp minima. Adaptive optimizers such as Adam and AdaGrad further alter the optimization trajectory by using different learning rates for different parameters, thereby producing different implicit regularization effects.

The concept of implicit regularization describes the intrinsic preference of the optimization algorithm itself over the solution space, rather than constraints imposed by explicitly adding penalty terms such as L1/L2. Early studies (such as Gunasekar et al., 2017) proved on matrix factorization problems that gradient descent tends to converge to solutions with minimum nuclear norm; on linear models, SGD tends to find solutions with minimum L2 norm. For deep nonlinear networks, the exact form of implicit regularization remains not fully understood, but numerous experiments show that hyperparameters such as batch size, learning rate, and optimizer choice all affect which class of solution the model ultimately converges to, and thereby its generalization performance. This suggests that optimization and generalization are not independent problems—the choice of optimization algorithm is itself a form of implicit inductive bias.

Major Schools of Deep Learning Theory

Current deep learning theory research can be broadly divided into the following directions, each approaching this complex problem from a different angle.

Neural Tangent Kernel (NTK) Theory

Neural Tangent Kernel theory approximates the training process of infinitely wide neural networks as a kernel method, thereby transforming the non-convex deep learning problem into an analyzable linear one. This framework provides a powerful tool for understanding training dynamics, but its limitations are equally apparent—real networks are not infinitely wide, and NTK struggles to fully capture the essence of feature learning.

To understand NTK theory, we first need to understand the basic idea of Kernel Methods. Kernel methods are a mature class of techniques in machine learning, exemplified by support vector machines (SVM). A kernel function k(x, x') measures the similarity between two inputs, using the "kernel trick" to implicitly map data into a high-dimensional or even infinite-dimensional feature space, where linear classification or regression is performed—thereby achieving nonlinear decision boundaries in the original space. Kernel methods come with complete theoretical guarantees: a closed-form global optimum exists, there are rigorous generalization bounds, and the training process is a convex optimization problem. NTK leverages precisely this mature toolkit—by proving that infinitely wide neural networks are equivalent to a kernel method, researchers can bring the analysis of deep learning into an existing mathematical framework.

NTK was formally introduced by Jacot, Gabriel, and Hongler in their 2018 NeurIPS paper. Its core idea is that as the width of a neural network approaches infinity, the network parameters barely move during gradient descent training, and the network's function output can be precisely described by a fixed kernel function—the NTK. The training dynamics degenerate into a kernel regression problem with a global linear convergence guarantee. The elegance of this framework lies in transforming the highly nonlinear, non-convex deep learning problem into a kernel method with mature analytical tools, thereby rigorously proving the global convergence of infinitely wide networks under gradient descent. However, real networks have finite width, and the "feature learning" phenomenon observed in practice—where network representations change significantly during training—is precisely what the NTK framework cannot capture. This is the core motivation behind current theoretical research seeking to go beyond NTK.

Loss Landscapes and Optimization Geometry

Another research direction focuses on the geometric structure of the loss function. Research shows that although the loss surface of deep networks is highly non-convex, its local minima are often of similar quality and connected by vast "flat regions." Flat minima typically correspond to better generalization performance, providing an intuitive geometric explanation for training stability.

Hochreiter and Schmidhuber first proposed in 1997 the intuition that flat minima are associated with better generalization: a solution found in a "wide valley" of the loss surface is insensitive to small perturbations of the parameters, which correlates with stable model performance on new data. In their 2017 experimental study, Keskar et al. further showed that large-batch training tends to converge to sharp minima, corresponding to worse generalization, while small-batch training tends toward flat minima. However, this view has also been challenged: Dinh et al. proved that any minimum can be made flat or sharp through reparameterization, indicating that simple curvature measures may be insufficient to explain generalization. How to precisely define and measure "flatness" remains an open question, and related research has given rise to practical optimization algorithms such as Sharpness-Aware Minimization (SAM).

The Information Bottleneck: An Information-Theoretic Perspective

The information-theoretic framework represented by the Information Bottleneck attempts to understand deep learning from the perspective of data compression. This theory holds that during training, a network first memorizes input information, then progressively compresses it layer by layer, discarding irrelevant details and ultimately retaining the representation most useful for the task. This view is quite appealing but remains controversial in academia.

The information bottleneck framework was originally proposed by Tishby and Pereira in 1999 to describe the tradeoff between optimal data compression and prediction. Tishby and Schwartz-Ziv introduced it into deep learning in 2017, proposing that neural network training has two phases: first empirical risk minimization (the fitting phase), then information compression (the forgetting phase), during which the network gradually discards input information irrelevant to the labels. However, researchers such as Saxe soon raised systematic challenges—they found that the compression phase is not universal, and its appearance depends heavily on the choice of activation function (e.g., compression often does not occur when using ReLU) and on how mutual information is estimated. Information bottleneck theory remains a controversial open question to this day, but it has advanced the overall research direction of understanding representation learning with information-theoretic tools.

The Persistent Gap Between Theory and Practice

You may not have noticed, but despite the continual advance of theoretical research, a clear gap remains between theory and engineering practice. The vast majority of breakthrough model architectures and training techniques—including the Transformer, residual connections, and batch normalization—were first proven effective in practice, with theoretical explanations often lagging years behind.

The Transformer architecture was proposed by Vaswani et al. in 2017. Its core component, the self-attention mechanism, models long-range dependencies by dynamically computing a weight matrix across all positions in a sequence, completely replacing the previously dominant recurrent neural networks (RNNs). Residual connections were introduced by He et al. in the 2015 ResNet paper, mitigating the vanishing gradient problem in deep networks by adding the layer input directly to the layer output (a skip connection). Batch Normalization was proposed by Ioffe and Szegedy in 2015, stabilizing training by normalizing activations over each mini-batch. All three techniques were confirmed effective in practice before theoretical explanations arrived: the optimization-geometry explanation for residual connections was only gradually established years later, and the generalization properties of the Transformer remain an active research topic.

This "practice first" pattern is especially pronounced in AI. On the one hand, it demonstrates deep learning's powerful empirically driven nature; on the other, it exposes the inadequacy of existing theoretical tools: we still lack a unified framework to predict why a given architecture will succeed.

Why Theoretical Research Is Indispensable

Some question whether theoretical research is still necessary, given how successful deep learning practice has been. The answer is yes.

Improving efficiency: Solid theory can guide more efficient model design, reducing the waste of computing power caused by blind trial and error.
Ensuring safety: As AI systems are deployed in high-stakes domains such as healthcare, finance, and autonomous driving, guarantees of model interpretability and safety are indispensable—and these are inseparable from theoretical support.
Advancing science: Understanding the nature of intelligence is itself a profound scientific question, and deep learning theory may bring new insights to cognitive science and neuroscience.

Conclusion: The Unsolved Black Box and Future Directions

Deep learning theory remains an open and active research frontier. We have accumulated a great many fragmentary insights—from NTK to implicit regularization, from loss landscapes to the information bottleneck—but have yet to form a complete, unified theoretical system.

In the era of large models, this theoretical absence has become increasingly pressing. The phenomenon of emergent capabilities—where large language models suddenly exhibit previously unobserved abilities on specific tasks once the parameter scale crosses a certain threshold, such as few-shot reasoning, code generation, and chain-of-thought reasoning—poses a particularly severe challenge to theoretical research. Wei et al. systematically documented this phenomenon in 2022, finding that certain abilities are nearly zero in small models but rise sharply in large ones, exhibiting discontinuous phase-transition characteristics.

It is worth noting that this "phase transition" metaphor has a precedent in physics: phase transitions in statistical physics describe the discontinuous, abrupt changes in a system's macroscopic properties as its parameters cross a critical point, such as water freezing or the magnetization transition of magnetic materials. However, Schaeffer et al. raised an important counterargument in 2023: they argue that so-called "emergence" is largely an artifact of the choice of evaluation metric—when using a coarse-grained discrete metric (such as whether an answer is correct), smooth performance gains appear as abrupt jumps; switching to a continuous metric (such as log probability) often reveals a smooth curve. This debate reminds us that identifying genuine emergence requires more rigorous experimental design, and it reflects the methodological challenges current theoretical tools face when confronting ultra-large-scale models. Most existing generalization theories assume performance varies smoothly with scale and cannot predict or explain such discontinuous leaps, revealing the fundamental limitations of current theoretical tools in the face of ultra-large-scale models.

As model scales continue to swell and emergent capabilities keep appearing, we need all the more to answer that fundamental question: How exactly do these black boxes work? Perhaps the next major breakthrough in deep learning will come precisely from cracking these theoretical puzzles.