Freeze the Decoder, Fix Only the Encoder: An Efficient Adaptation Method for SVD-Based KV Cache Compression

A Overlooked Experimental Pitfall

When comparing parameter-efficient fine-tuning (PEFT) approaches, researchers often run all baselines with a single shared learning rate. It sounds fair — but it's a trap. This arXiv paper exposes a common yet underappreciated methodological flaw: when the methods being compared have vastly different numbers of trainable parameters, using a uniform learning rate simultaneously suppresses the mean performance of high-parameter methods while inflating their variance. The result is an artificially constructed advantage that consistently lands on the method with the fewest parameters.

The paper examines this problem in a concrete setting: post-hoc SVD-based KV cache compression — an increasingly relevant direction in practice, since KV cache memory consumption is one of the primary bottlenecks in long-context inference.

How SVD-Based KV Cache Compression Works

To understand this work, we first need to understand the technical context. Given a pretrained model, you can apply Singular Value Decomposition (SVD) to its Key and Value weight matrices, factorizing them into two parts: a dimensionality-reducing projection (the "encoder") and a dimensionality-expanding projection (the "decoder"). This effectively transforms standard attention into a low-rank cache structure reminiscent of Multi-Head Latent Attention (MLA).

Low-rank decomposition inevitably introduces truncation error, causing a drop in model accuracy. To recover this loss, a brief fine-tuning pass is required — what the paper calls "healing." The central question then becomes: which parameters should be updated during healing? Only the encoder, only the decoder, or both?

SVD (Singular Value Decomposition) is a classical matrix factorization method in linear algebra: any matrix W can be decomposed as U·Σ·Vᵀ, where Σ is a diagonal matrix with singular values sorted in descending order. The key idea behind low-rank approximation is to retain only the top r singular values, replacing the original large matrix with the product of two narrower matrices — this is precisely the origin of the "encoder-decoder" structure. In the KV cache setting, a higher compression ratio means a smaller rank r, less cache memory, but more truncated information and greater performance degradation. Multi-Head Latent Attention (MLA), introduced in the DeepSeek^Ad model series, shares the same core idea: projecting KV into a low-dimensional latent space for caching, then decoding back to high dimensions at inference time — achieving near-full attention expressiveness at a fraction of the memory cost. SVD post-processing applies this philosophy retroactively to existing pretrained models, without requiring training from scratch.

The "False Victory" Under a Shared Learning Rate

Under the shared learning rate setting, the experimental results look remarkably clear: freezing the decoder and healing only the encoder substantially outperforms healing the decoder alone or healing both. Fewer parameters, better performance — nearly the ideal result any PEFT paper would dream of.

But the authors didn't stop there. They tuned a dedicated learning rate for each baseline individually, then ran three random seeds per configuration. The results reversed dramatically — the "clear advantage" vanished. Healing only the encoder didn't actually outperform the other approaches; instead, it reached parity with them.

In other words, the previously observed advantage was entirely an artifact of the shared learning rate experimental setup. High-parameter methods were suppressed by an ill-suited learning rate, making the low-parameter method appear to stand head and shoulders above the rest.

The intuition behind why learning rate affects parameter groups of different sizes has a clean optimization-theoretic explanation: methods with more trainable parameters face a more complex loss landscape in higher-dimensional space, requiring smaller learning rates for stable convergence. Methods with very few parameters correspond to a lower-dimensional optimization problem, and the same learning rate is often already too large — yet they can "stumble" their way to decent results. When researchers choose a learning rate friendly to the small-parameter method, the large-parameter method is effectively trained at an inflated learning rate, causing more oscillation in gradient updates and suppressed performance and stability. This bias is not random noise — it systematically favors the method with the fewest parameters and will persist regardless of how many experiments are run, unless each method's optimal learning rate is searched independently.

Where the Real Value Lies

After the reversal, the value of this research doesn't disappear — it actually becomes more grounded. While healing only the encoder achieves mere parity in accuracy, it delivers tangible resource savings:

• 3× fewer trainable parameters
• 3× less optimizer state memory

This makes encoder-only healing a "low-memory, plug-and-play" solution, particularly well-suited for retrofitting pretrained models with low-rank KV cache compression during training. Same accuracy, lower cost — that's a genuinely meaningful engineering contribution, not a phantom accuracy advantage manufactured through experimental design.

Rigor of Validation

The paper's conclusions rest on a reasonably solid experimental foundation. The researchers validated the parity result on the vision-language model Qwen2.5-VL-3B-Instruct, using per-method learning rate tuning and three-seed repetition for a single compression ratio covered by the protocol. To strengthen generalizability, they also reproduced the same phenomenon on text-only benchmarks across two different backbone models.

It's worth noting objectively that the paper explicitly states the validation covers only "a single compression ratio within the scope of this protocol" — this is a notable scope limitation. Whether the findings hold consistently across different compression ratios remains an open question.

A Warning for the Entire Fine-Tuning Research Community

The paper's greatest contribution may not lie in KV cache compression itself, but in the methodological warning it raises: any study comparing fine-tuning approaches with different numbers of trainable parameters should be on guard against the shared learning rate trap.

In an era where PEFT methods, LoRA variants, and various adapter approaches proliferate, many papers' "SOTA" conclusions are built on cross-method comparisons. If those comparisons don't independently tune the learning rate for each method, then some portion of the "fewer parameters, better performance" conclusions may need to be revisited. This is a low-cost, high-impact methodological correction that every researcher running fine-tuning experiments should keep in mind.

LoRA (Low-Rank Adaptation) is currently one of the most widely used PEFT methods. Its core idea is to add a low-rank matrix ΔW = A·B in parallel with the pretrained weights, training only the small matrices A and B. The "rank" in LoRA directly determines the number of trainable parameters, and different rank choices can lead to parameter counts differing by several to tens of times. The shared learning rate trap is equally prevalent in comparative experiments across LoRA and its many variants (e.g., AdaLoRA, DoRA, VeRA). If a paper claims a method "achieves higher accuracy with fewer parameters" but the experimental section doesn't explicitly state that learning rates were searched independently for each baseline, that conclusion is at risk of being contaminated by the bias exposed in this paper. The check is simple: look at the hyperparameter search section and confirm that every baseline has an independent learning rate tuning record along with a sufficient number of random seed repetitions.