Neural Expansion

A recent study titled "Neural Expansion: A Unified Mechanism for How Deep Neural Networks Generalize" proposed "neural expansion," a unified geometric framework that explains generalization from the perspective of directional coverage in input space.

The neural expansion framework shifts the focus from parameter space to the geometric structure of input space, proposing that the training process leads to a gradual expansion of "direction-selective ray coverage" in input space, thereby forming stable prediction regions around training samples.

Through empirical studies on different architectures (such as fully connected networks, convolutional networks, and residual networks) and datasets (such as MNIST, CIFAR-10, and ImageNet subsets), the authors found that the training process continuously expands ray coverage, especially in directions where the Jacobian singular values are smaller.

Experiments show that the geometric structure of ray coverage can explain a large portion of correctly classified test inputs, as well as the prediction behavior on adversarial examples, mislabeled samples, and even out-of-distribution (OOD) samples.

Adversarial examples often lie at the edge of or outside ray coverage, while mislabeled samples may be covered within the expanded regions of training samples, causing the model to "confidently" give wrong predictions.

OOD samples may fall outside the coverage, thereby exhibiting high uncertainty.

The neural expansion framework explains OOD samples as inputs that fall outside the ray coverage of training samples, so the model's predictions on them lack stability and may exhibit high uncertainty or random predictions.

This framework is complementary to, rather than a replacement for, existing global theories (such as margin bounds): global theories provide necessary conditions or upper bounds for generalization, while neural expansion explains from the perspective of training dynamics how generalization is concretely realized.

The neural expansion framework may inspire new regularization methods and model design principles, for example, improving generalization by encouraging coverage expansion in low-curvature directions.

This research is still in an early stage, and future work can further explore how to apply neural expansion to more complex tasks (such as object detection and language models) and how to combine it with areas such as adversarial robustness and continual learning.