AttSVD
A KV cache compression method based on attention-guided singular value decomposition. It performs online truncated SVD for each prompt to achieve low-rank feature compression, reducing KV cache memory by approximately 50% without retraining while preserving model accuracy.
Timeline (last 90 days)
AttSVD is a training-free KV cache compression scheme that performs low-rank compression of the KV cache along the feature axis via attention-guided SVD.
AttSVD achieves accuracy comparable to the full dense cache while using at most 50% of KV cache memory across multiple models and benchmarks.
The paper "AttSVD: Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD" was published on arXiv.
AttSVD retains every token and compresses the representational dimensionality of each token vector along the feature axis through low-rank decomposition, avoiding information loss at the sequence level.
The basis used by AttSVD for low-rank compression is not fixed but is derived from each prompt's own attention geometry, retaining only the directions actually read by attention via online per-prompt truncated SVD.
AttSVD's persistent per-head KV memory footprint decreases proportionally with the retained rank.
AttSVD proposes two decoding cache strategies for different generation lengths: the accumulative mode for short generation scenarios and the streaming mode for long generation scenarios.
AttSVD introduces a per-matrix energy rule that independently determines sizes for logit space and attention quality, dynamically adjusting the compression ratio.
AttSVD's evaluation covers one agent benchmark and the complete LongBench long-context evaluation suite.
All Facts (9)
AttSVD's evaluation covers one agent benchmark and the complete LongBench long-context evaluation suite.
50%UnverifiedAttSVD achieves accuracy comparable to the full dense cache while using at most 50% of KV cache memory across multiple models and benchmarks.
50%UnverifiedAttSVD is a training-free KV cache compression scheme that performs low-rank compression of the KV cache along the feature axis via attention-guided SVD.
50%UnverifiedAttSVD retains every token and compresses the representational dimensionality of each token vector along the feature axis through low-rank decomposition, avoiding information loss at the sequence level.
50%UnverifiedThe basis used by AttSVD for low-rank compression is not fixed but is derived from each prompt's own attention geometry, retaining only the directions actually read by attention via online per-prompt truncated SVD.
50%UnverifiedAttSVD's persistent per-head KV memory footprint decreases proportionally with the retained rank.
50%UnverifiedAttSVD proposes two decoding cache strategies for different generation lengths: the accumulative mode for short generation scenarios and the streaming mode for long generation scenarios.
50%UnverifiedAttSVD introduces a per-matrix energy rule that independently determines sizes for logit space and attention quality, dynamically adjusting the compression ratio.
50%UnverifiedThe paper "AttSVD: Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD" was published on arXiv.
50%