AttSVD: Attention-Guided SVD Compresses KV Cache to Half the Memory Without Accuracy Loss

The Memory Bottleneck of Long Contexts: Why KV Cache Becomes a Roadblock

Autoregressive Transformers cache the key-value pairs (KV) for every layer and every attention head during inference to avoid redundant computation. The problem is that this KV cache grows linearly with context length. When models handle long documents, multi-turn conversations, or agentic tasks, the KV cache often becomes the single largest consumer of GPU memory, directly capping both the maximum context window and inference throughput.

A newly published arXiv paper, AttSVD: Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD, addresses this pain point with a training-free KV cache compression scheme. Across multiple models and benchmarks, it achieves performance on par with full-cache inference using as little as 50% of the original memory.

A Different Dimension for Saving Memory: From "Token Eviction" to "Feature Compression"

Most training-free KV cache optimization methods follow an "eviction" route — they estimate which tokens are unimportant and simply discard them. The fundamental issue with this approach is that it operates as an irreversible operation along the sequence axis: once a token is dropped, the information it carries is gone forever. If subsequent generation happens to need that information, the model will fail.

AttSVD takes a completely different approach. It keeps every token, but stores them more cheaply along the feature axis. Instead of deciding which tokens to keep or discard, it compresses the representational dimensionality of each token vector via low-rank decomposition. This strategy avoids the risk of sequence-level information loss — a gentler and more reversible form of compression.

Core Mechanism: Online, Per-Prompt Attention-Guided SVD

AttSVD's key innovation is that the basis used for low-rank compression is not fixed — it is derived from the attention geometry of each individual prompt. Specifically, it performs an online, per-prompt truncated SVD, retaining only the directions that attention actually "reads."

The direct benefit: the per-head KV memory footprint decreases proportionally to the retained rank. And because the compression directions come from the geometry of attention itself, they are inherently interpretable — what is preserved is precisely the information subspace that attention actually consumes.

Singular Value Decomposition (SVD) is a classic matrix factorization method in linear algebra: any matrix $M$ can be decomposed as $M = U\Sigma V^T$, where $\Sigma$ is a diagonal matrix with singular values sorted in descending order. Truncated SVD retains only the top $r$ largest singular values and their corresponding left and right singular vectors, approximating the original matrix with a rank-$r$ low-rank matrix, thereby achieving information compression. The lower the rank $r$, the smaller the storage cost — but the larger the approximation error. AttSVD's key distinction is performing this decomposition online and per-prompt: rather than precomputing a fixed global projection basis offline, it computes the SVD of each input's KV matrix at inference time, capturing the attention geometry specific to that prompt. This means compression directions dynamically adapt to each input's information distribution, rather than using a static basis that is merely average-optimal across all inputs — which is precisely what "prompt-adaptive" means.

Mainstream KV cache eviction methods (such as H2O, ScissorHands, StreamingLLM, etc.) operate by assigning each token an importance score (typically based on cumulative attention weights), then permanently removing the KV vectors of low-scoring tokens from the cache. These methods sparsify along the sequence length dimension, yielding intuitive memory savings — but at the cost of irreversible decisions: evicted tokens are permanently absent from all subsequent generation steps. AttSVD's "feature-axis compression" route, by contrast, keeps all tokens while dimensionality-reducing the K and V vectors themselves: vectors originally of dimension $d_{head}$ are projected into a lower-dimensional subspace for storage, then projected back to the original space for attention computation. This represents a fundamentally different pattern of information loss: the former completely eliminates contributions from certain sequence positions; the latter preserves approximate contributions from all positions via lossy compression.

Two Decoding Cache Strategies: Covering Both Short and Long Generation

The paper proposes two decoding-time cache strategies tailored to different generation lengths:

• Accumulating: suited for short generation scenarios;
• Streaming: suited for long generation scenarios.

This distinction matters. Short and long generation have very different cache read/write patterns, and a single fixed strategy cannot simultaneously optimize for both efficiency and accuracy. By offering two modes, AttSVD lets practitioners choose the most appropriate cache management approach based on their actual task characteristics.

Two Refinements That Make Compression Adaptive

AttSVD introduces two additional design choices that make compression smarter:

Per-matrix energy rule: It independently determines sizing for logit space and attention mass. This means compression rates are not one-size-fits-all — they are dynamically adjusted based on each matrix's energy distribution, preventing over-compression in information-dense regions.

Attention-aware basis truncation: It truncates only within the space that attention actually reads, thereby preserving both the attention logits and the attention output simultaneously. This ensures compression does not damage the critical components of the attention computation.

Worth noting: these decomposition factors also provide "free" per-head interpretability insights — researchers can observe the effective rank of each attention head and the geometry that attention consumes, which is also valuable for understanding Transformer internals.

Empirical Results: Half the Memory, No Accuracy Drop

The paper validates AttSVD across multiple models, with evaluations covering one agentic benchmark and the full LongBench long-context evaluation suite. Results show that AttSVD performs on par with the full dense cache while using at most 50% of KV cache memory.

The significance of this result is that it demonstrates the viability and practicality of feature-axis compression: cutting long-context inference memory costs in half without sacrificing model capability. For deploying long-context LLMs, running agentic tasks, or pursuing higher concurrent throughput, this class of solutions offers direct engineering value.

LongBench is the most widely used comprehensive evaluation benchmark in the long-context understanding domain, covering subtasks including single-document QA, multi-document QA, summarization, few-shot learning, code completion, and synthetic tasks. Document lengths commonly exceed 8K tokens, with some tasks exceeding 32K tokens. It requires models to genuinely "understand" and utilize long-range context information rather than relying on proximal information or parametric memory — making it a rigorous test of a KV cache compression scheme's information retention capability. Agentic benchmarks test model performance across multi-step tool calls, planning, and feedback loops; such tasks typically have longer and more structurally complex conversation histories than ordinary QA, placing higher demands on complete preservation of information from early turns in the cache. AttSVD maintaining performance on par with the full cache across both of these very different evaluation settings demonstrates strong cross-task generalization of its compression strategy.

Summary

AttSVD represents a new direction in KV cache optimization: rather than debating "which token to delete," it asks "how to store each token more efficiently." By leveraging the geometric structure of attention itself for adaptive low-rank decomposition, it preserves all sequence information while halving memory usage. Combined with accumulating and streaming decoding strategies, along with the per-matrix energy rule and attention-aware basis truncation refinements, this approach demonstrates effectiveness on long-context and agentic benchmarks. As a training-free, interpretable solution, it offers a compelling technical path for deploying long-context inference in production.