Finite-Rank Bottleneck: A New Theoretical Perspective on LoRA Low-Rank Adaptation

Why does low-rank updating work? An explanation without linearization

Low-Rank Adaptation (LoRA) has become the standard method for parameter-efficient fine-tuning of large models, but a complete theoretical explanation of "why low-rank updates are sufficient" has always been lacking. Most analyses rely on the Neural Tangent Kernel (NTK) framework, which linearizes the network around the pretrained weights. However, linearization itself is an approximation and may obscure the true mechanism by which LoRA operates.

A recent study proposes a completely different path: rather than relying on linearization, it starts from the structure of the frozen backbone network itself. Its core observation is that—once the backbone is frozen, the loss function can only "see" each adapter weight matrix through the finite-rank linear map induced by the frozen activations. In other words, the influence of each LoRA module on the output is compressed into a low-dimensional subspace, and the rank of this subspace is called the "bottleneck rank."

The key value of this perspective is that it explains why LoRA's rank is determined by the model structure rather than by sample size. In traditional generalization theory, rank upper bounds are usually tied to the number of training samples; here, the bottleneck rank becomes the dominant factor controlling the low-rank solution.

Neural Tangent Kernel (NTK) and linearization approximation

The Neural Tangent Kernel (NTK) is a tool for analyzing the training dynamics of neural networks. Its core idea is that as network width tends to infinity, the training dynamics of the network near initialization can be approximated by a linear model whose kernel function is the NTK. Based on this, many theoretical works linearize the behavior of pretrained models during fine-tuning, i.e., they approximate the network output using a first-order Taylor expansion, thereby transforming the complex non-convex optimization problem into a convex kernel method problem. However, this linearization holds only for small weight updates, whereas LoRA's adapter matrices, although low-rank, may not have updates small enough for the approximation to be valid; thus linearization may fail to capture LoRA's true mechanism. The research introduced in this article precisely avoids this approximation and analyzes directly from the structure of the frozen backbone.

Trace-norm regularization: forcing the optimizer to choose low-rank solutions

The study further introduces trace-norm regularization and proves that among all matrices producing the same observed output, the optimizer tends to choose low-rank matrices. This is no coincidence—the trace norm is a convex relaxation of matrix rank, and under the constraint of a finite-rank linear map, minimizing the trace norm naturally leads to low-rank solutions.

The resulting global minimizer has its rank controlled by the bottleneck rank rather than by sample size. This conclusion provides more direct theoretical support for the practical effectiveness of LoRA: low rank is not an artificial constraint but is determined by the intrinsic structure of the loss landscape under the frozen backbone.

A tighter upper bound in multi-head attention

For multi-head attention, the study exploits two additional invariances to obtain a tighter rank upper bound:

• Shift invariance of row-wise softmax: Adding an arbitrary constant to each row of the score matrix leaves the output unchanged after softmax. This means there are redundant directions in the score matrix that can further compress the effective rank.
• Observability of value updates: Updates to the value matrix can only be observed after attention weighting and the frozen output projection; this structural constraint likewise lowers the upper bound on the observable rank.

Together, these two points make the LoRA rank upper bound in the multi-head attention setting tighter than in the general case, and also explain why in practice the LoRA rank of attention layers can often be set lower without loss of performance.

Trace norm and nuclear norm

The trace norm, also known as the nuclear norm, is the sum of a matrix's singular values. It is a convex relaxation of matrix rank: under a unit-ball constraint, minimizing the trace norm tends to produce low-rank matrices, because the smaller the trace norm, the more the singular values concentrate in a few directions. This property makes the trace norm a commonly used regularizer in machine learning and signal processing to promote low-rank solutions. In the context of this article, trace-norm regularization is used to prove that among all possible weight matrices producing the same observed output, the optimizer prefers solutions with smaller trace norm, i.e., low-rank solutions. Since the trace norm is convex, this analysis bypasses the difficulties of non-convex optimization and provides structural guarantees for the global optimum.

Shift invariance of row-wise softmax

The shift invariance of row-wise softmax means that for each row of the attention score matrix, adding an arbitrary constant vector (i.e., adding the same constant to all elements in a row) leaves the output after the softmax function unchanged. This is because softmax exponentiates the inputs and then normalizes; the constant factor cancels in the numerator and denominator. This property implies redundant degrees of freedom in the score matrix—changes along the all-ones direction do not affect the final output. Therefore, when analyzing LoRA's effective influence on attention layers, these redundant directions can be compressed, thereby lowering the upper bound on the observable rank. Using this invariance, theoretical analysis can give tighter rank constraints than in the general case, explaining why the LoRA rank of attention layers can be set lower.

The dual conclusion on the non-convex side: stationary points must be rank-deficient

The other half of the theoretical analysis focuses on non-convex optimization. For the factorized LoRA objective with weight decay, the study proves that after an arbitrarily small generic positive semidefinite perturbation, as long as the LoRA rank exceeds the threshold determined by the bottleneck rank, every first-order stationary point is rank-deficient.

The significance of this result is that it demonstrates the inevitability of low-rank solutions from the perspective of optimization dynamics—not only is the global optimum low-rank, but even local stationary points cannot maintain full rank. This complements the conclusion on the trace-norm side: one starts from a convex relaxation, the other from non-convex stationary points, and both point to the same conclusion.

The study also proves that the global minimizer of the perturbed objective is approximately optimal for the original trace-norm problem, and provides a uniform generalization bound controlled by the spectral structure of the frozen features. This means that generalization performance is directly related to the spectral properties of the frozen features, rather than depending solely on the rank of the adapter matrix.

First-order stationary points and rank deficiency

A first-order stationary point is a point where the gradient is zero, including local minima, local maxima, and saddle points. In non-convex optimization, first-order stationary points are not necessarily optimal solutions, but analyzing their properties helps in understanding the optimization landscape. Rank deficiency means that the rank of a matrix is less than its maximum possible value (i.e., the smaller of the number of rows or columns). This article proves that for the factorized LoRA objective, after applying an arbitrarily small generic positive semidefinite perturbation, as long as the LoRA rank exceeds the threshold determined by the bottleneck rank, every first-order stationary point is rank-deficient. This means that the optimization process cannot maintain a full-rank adapter matrix; low-rank solutions are not only globally optimal but also the inevitable structure of local stationary points. This conclusion reinforces the effectiveness of low rank from the non-convex side.

Implications for LoRA practice

These theoretical results have several actionable implications for practical hyperparameter tuning:

1. Rank selection should refer to the bottleneck rank: LoRA rank is not better when larger; beyond the bottleneck rank threshold, additional rank may not bring observable output changes and may instead increase optimization difficulty.
2. Attention layers can be reduced in rank more aggressively: The shift invariance of softmax and the observability constraint of value updates provide tighter rank upper bounds for attention modules.
3. The role of weight decay is redefined: In factorized LoRA, weight decay is not only a regularization method but is also directly related to the rank-deficiency property of stationary points, affecting the structure of the solution the optimization ultimately falls into.
4. Generalization depends on the feature spectrum: The spectral structure of the frozen features determines the generalization bound, suggesting that feature quality deserves more attention than adapter capacity.

Conclusion

The value of this work lies not in overturning NTK analysis but in providing an alternative path that does not rely on linearization. The concept of the finite-rank bottleneck advances the effectiveness of LoRA's low rank from "empirical observation" to "structural necessity," and gives a more refined characterization of the rank upper bound for multi-head attention. For those engaged in research and engineering practice of parameter-efficient fine-tuning, the bottleneck rank is a new variable worth adding to the intuition toolbox.

One-line Summary

Under a frozen backbone, LoRA's low-rank effectiveness is determined by the bottleneck rank rather than sample size, and stationary points are necessarily rank-deficient.

Paragraph Summary

This study proposes a new theoretical framework that does not rely on linearization to explain why low-rank adaptation (LoRA) works. The core finding is that after freezing the backbone, the loss function observes the adapter weights only through a finite-rank linear map, whose rank—called the bottleneck rank—determines the rank upper bound of the low-rank solution rather than the number of training samples. Through trace-norm regularization, it is proved that the optimizer prefers low-rank matrices among equivalent outputs; in multi-head attention, tighter upper bounds are obtained using softmax shift invariance and the observability of value updates. Non-convex analysis further shows that when the LoRA rank exceeds the bottleneck rank threshold, first-order stationary points are necessarily rank-deficient under arbitrarily small perturbations. The generalization bound is controlled by the spectral structure of the frozen features. Practical implications include: rank selection should refer to the bottleneck rank, attention layers can be reduced in rank more aggressively, weight decay affects the solution structure, and feature spectrum quality is more critical than adapter capacity.