Transfer Learning Breaks the Scaling Law: Ridge Regression Achieves Minimax Optimality

Can transfer learning deliver substantive performance gains? Especially with respect to scaling laws, this is a key question in the era of large models. A recent study, Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning, theoretically proves that transfer learning can achieve the minimax-optimal scaling exponent in a canonical high-dimensional regression setting, providing a solid theoretical foundation for the reuse of pretrained models.

Research Background and Problem Setup

Pretrained models are often reused for downstream tasks, but can this transfer learning process yield significant improvements in scaling laws? The study focuses on a prototypical high-dimensional regression setting: the source model is trained via ridge regression, and the target model is fine-tuned with (L_2) distance regularization. This framework captures the core of transfer learning—leveraging knowledge from the source task to improve performance on the target task.

The researchers examine the scaling behavior of risk, i.e., how model performance changes as the amount of data or dimensionality increases. Scaling laws describe this pattern of change, and the minimax-optimal exponent represents the best possible rate theoretically achievable.

Minimax-optimal exponent: In statistical learning theory, the minimax-optimal exponent describes the optimal error convergence rate that any estimation method can achieve across all possible data-generating distributions and in the worst case. Specifically, researchers consider a class of parameters or functions and examine the maximum risk between all estimators and the true parameter; the minimax-optimal exponent is the rate at which this maximum risk tends to zero fastest as sample size or dimensionality grows. If a method achieves this exponent, it means it is optimal even in the worst case and cannot be improved in order by other methods. In this paper, the concept is used to measure whether the scaling rate of transfer learning risk with data size or dimensionality reaches the theoretical limit.

Deterministic Equivalence and Scaling Laws

The study first establishes a deterministic equivalent for the risk of the two-step process, covering both model shift and covariance shift. The deterministic equivalent is a powerful tool that approximates complex stochastic risk with a deterministic expression, facilitating analysis of scaling behavior.

By specializing the deterministic equivalent to the setting of a shared regression function and an exchangeable power-law covariance spectrum, the researchers identify a specific regime in which the scaling law achieves the minimax-optimal exponent. This means that, under certain conditions, transfer learning can achieve the theoretically best performance scaling.

Interestingly, in this optimal regime, the optimal choice of source regularization depends only on the source data and is independent of the target task. This indicates that source regularization is universal and can generalize across different target tasks. This finding simplifies hyperparameter tuning for transfer learning, as there is no need to re-tune source regularization for each target task.

Deterministic equivalent: The deterministic equivalent is an approximation tool widely used in random matrix theory and high-dimensional statistics. Its core idea is that, for a certain stochastic risk function (such as the prediction error of ridge regression), one can construct a deterministic expression that depends only on data distribution parameters (such as the covariance spectrum and noise variance), such that when dimensionality and sample size tend to infinity proportionally, the difference between the stochastic risk and this deterministic expression converges to zero in probability. In this way, the originally complex stochastic analysis is transformed into the analysis of a deterministic function, greatly simplifying the derivation of scaling behavior. This paper uses the deterministic equivalent to precisely characterize the risk of the two-step transfer learning process and, on this basis, identifies the regime that achieves the minimax-optimal exponent.

Fine-Tuning and Stochastic Gradient Descent

The study further shows that fine-tuning via one-pass stochastic gradient descent (SGD) can also achieve the same optimal rate. This facilitates practical applications, as SGD is one of the most commonly used optimization algorithms in deep learning. The theoretical analysis is supported by synthetic experiments and language model experiments, enhancing the reliability of the conclusions.

One-pass stochastic gradient descent (one-pass SGD): One-pass stochastic gradient descent refers to traversing the training set only once, using one sample (or mini-batch) at a time to compute the gradient and update parameters. Compared with multi-epoch SGD, one-pass SGD has lower computational overhead and is closer to online learning scenarios. Under the theoretical framework of this paper, the researchers prove that starting from the source model, only one pass of SGD fine-tuning on the target task data is needed to achieve the same minimax-optimal scaling rate as ridge regression fine-tuning. This provides theoretical support for practical large-model fine-tuning: complex multi-epoch optimization or elaborate hyperparameter tuning is unnecessary; simple one-pass SGD can achieve optimal transfer performance.

Multi-Source Cross-Validation

According to analyses by Bilibili UP creators and YouTube channels, the theoretical results of this study corroborate empirical experiments. Synthetic experiments verify the accuracy of the deterministic equivalent, while language model experiments demonstrate the potential of transfer learning in practical tasks. Although different sources interpret specific experimental details slightly differently, the core conclusion is consistent: under appropriate conditions, transfer learning can achieve minimax-optimal scaling.

Implications for the AI Field

This study provides a rigorous theoretical foundation for transfer learning and proves its scaling advantages. In the era of large models, this means that the reuse of pretrained models is not only an engineering practice but also has theoretical guarantees of optimality. In the future, this framework may be extended to more complex models and tasks, further advancing the development of transfer learning.

One-line Summary

Theoretical proof that transfer learning can achieve the minimax-optimal scaling exponent in high-dimensional ridge regression.

Paragraph Summary

This paper addresses the core question of whether transfer learning can deliver substantive improvements in scaling laws, conducting a theoretical study in the prototypical setting of high-dimensional ridge regression. The authors first establish a deterministic equivalent for the risk of the two-step transfer learning process, covering both model shift and covariance shift, and then specialize it to the setting of a shared regression function and an exchangeable power-law covariance spectrum, identifying a regime in which transfer learning achieves the minimax-optimal scaling exponent. The study finds that in this optimal regime, the optimal choice of source regularization depends only on the source data and is independent of the target task, exhibiting cross-task universality. In addition, one-pass stochastic gradient descent fine-tuning can also achieve the same optimal rate, and the theoretical results are validated by synthetic experiments and language model experiments.