From Random Features to Fully Trained Networks: Phase Transitions in Neural Networks

In scientific computing, constructing accurate and computationally feasible approximation spaces is a fundamental task. Partially trainable neural networks serve as a bridge between fixed random features and fully adaptive representations—how does their performance change with the trainable proportion? A recent paper reveals phase transition patterns in approximation bounds and training error through theoretical analysis and experimental validation, offering a new perspective for understanding this transition.

Phase Transition Phenomenon: From Rapid Improvement to Diminishing Returns

The paper quantifies the phase transition pattern in approximation bounds and observes similar behavior in training error: an initial phase of rapid but smooth improvement, followed by diminishing returns, with sharper transitions when initialization mismatch is more severe. This phase transition does not occur abruptly; rather, the rate of performance improvement changes significantly as the trainable proportion increases. When the trainable proportion is small, adding a few trainable parameters leads to a substantial drop in the approximation bound; but once the proportion exceeds a certain threshold, further increases in trainable parameters yield limited benefits. This phenomenon is validated across multiple network architectures and settings, including networks with up to 15 layers.

Theoretical Framework: Explicit Expression of Approximation Rates

Under the assumptions of polynomial spectral decay (exponent α>1) and leverage compatibility, the paper establishes the squared L2 approximation rate for two-layer networks of width m and trainable proportion P: O((κ_P log(m/δ)/m)^α), where κ_P = (P/κ_1^* + (1-P)/κ_0^)^{-1}. This bound holds uniformly over the unit ball of the reproducing kernel Hilbert space (RKHS) with probability at least 1-δ. The constants κ_0^ and κ_1^* quantify the sampling mismatch of the fixed group and the trainable group, respectively, relative to the target RKHS. This theoretical result connects approximation error to factors such as the trainable proportion, network width, spectral decay exponent, and initialization quality, providing a mathematical foundation for understanding the phase transition.

The reproducing kernel Hilbert space (RKHS) is a class of Hilbert spaces defined by symmetric positive definite kernel functions, whose key property is that the evaluation functional at each point is continuous, ensuring well-defined pointwise evaluation. In kernel methods, the RKHS is often used as the hypothesis space, and its unit ball corresponds to the set of "smooth" functions of the kernel. In approximation theory, the target function is typically assumed to belong to some RKHS, and network approximation rates are measured as the worst-case error over the unit ball of that space. The "target RKHS" mentioned in the paper refers to the function space defined by the kernel corresponding to the network features, while κ_0^* and κ_1^* measure the "leverage" differences between the kernels induced by the fixed feature group and the trainable feature group and the target kernel.

Leverage compatibility is a concept in statistics used to analyze high-dimensional regression and sampling bias. In random feature models, each feature contributes differently to the target function; some features may be particularly important while others play a minor role. The leverage score measures the influence of an individual sample or feature in the kernel matrix. The leverage compatibility assumption requires that the sampling distribution maintains a certain consistency with the "important directions" of the target function in feature space, thereby preventing extreme sampling bias from causing approximation error explosion. Specifically in this paper, this assumption ensures that the random features of the fixed group and the trainable group can cover the target RKHS in a controlled manner, so that the constants κ_0^* and κ_1^* in the approximation rate bound are well-defined and independent of network width.

Concrete Validation: Spherical-Biased ReLU Features

For spherical-biased ReLU features of dimension d, the paper validates the above assumptions and provides specific parameters: when the initialization scale τ≥1, α = (d+2)/(d-1), κ_0^* = 4√2 τ(d+1)/d; under ridge leverage sampling, κ_1^* = 1. Taking d=5 and τ=10 as an example, a trainable proportion of approximately 4.1% captures 90% of the approximation bound reduction from P=0 to P=1. This means that training only a tiny fraction of parameters can achieve approximation capability close to that of a fully trained network, greatly improving computational efficiency. This result has important guiding significance for practical applications, especially in resource-constrained scenarios.

Spherical-biased ReLU features are a random feature model commonly used in theoretical analysis: the input vector is first normalized to the unit sphere, then projected through a random weight matrix with a bias added, and finally a ReLU activation function is applied. The bias term is typically sampled from a fixed distribution, with its scale τ controlling the activation threshold. This feature mapping corresponds to the hidden layer structure of a two-layer neural network, where the first layer weights are randomly initialized and fixed, and only the second layer (or part of the first layer) is trainable. The specific parameters given in the paper (such as α = (d+2)/(d-1)) are derived under the conditions of input dimension d and initialization scale τ≥1; these parameters directly determine how the approximation rate varies with network width m and trainable proportion P.

Deep Networks and Structured Examples

The paper also illustrates through a structured three-layer example how a small number of trainable first-layer neurons can leverage fixed downstream blocks through nonlinear composition. This structure shows that even when only part of the first layer is trainable, the network can achieve powerful representational capability through the nonlinear transformations of subsequent fixed layers. Numerical experiments cover various settings, including networks with up to 15 layers, all exhibiting similar phase transition phenomena in training error. These experiments not only validate the theoretical predictions but also reveal the universality of phase transition behavior across different depths and architectures.

Practical Significance and Future Directions

The discovery of phase transition phenomena has practical significance for the design and training of neural networks. It suggests that in certain cases, training all parameters is unnecessary to achieve performance close to full training, thereby saving computational resources. At the same time, the influence of initialization quality on the phase transition point emphasizes the importance of initialization strategies. Future research can further explore methods for predicting phase transition points in different architectures and tasks, and how to leverage this phenomenon to design more efficient partially trainable networks. In addition, the applicability of the assumptions in the theoretical framework (such as spectral decay and leverage compatibility) to real data is also worth further investigation.

In summary, this study reveals the phase transition behavior of approximation bounds and training error in partially trainable neural networks through theoretical analysis and experimental validation, providing new insights for understanding the transition between random features and fully trained networks.

One-line Summary

Approximation bounds and training error in partially trainable neural networks exhibit phase transitions, with a small number of trainable parameters approaching fully trained performance.

Paragraph Summary

The paper investigates how approximation bounds and training error in partially trainable neural networks vary with the trainable proportion, discovering a clear phase transition phenomenon: rapid performance improvement initially, followed by diminishing returns, with sharper transitions when initialization mismatch is more severe. Theoretically, under polynomial spectral decay and leverage compatibility assumptions, the squared L2 approximation rate for two-layer networks is derived as O((κ_P log(m/δ)/m)^α), where κ_P is determined by the trainable proportion and sampling mismatch constants. For spherical-biased ReLU features, the assumptions are validated with specific parameters, and numerical results show that only about 4.1% trainable parameters (d=5, τ=10) are needed to capture 90% of the approximation bound reduction. A structured three-layer example and numerical experiments with up to 15 layers further validate the universality of the phase transition behavior, providing a theoretical basis for efficient network design.