BasisKV
A low-rank KV cache compression method for distributed long-context LLM inference that jointly optimizes low-rank Value representations with the output projection, reduces tensor-parallel communication overhead, and reuses compressed Value caches to assist Key page routing.
Timeline (last 90 days)
BasisKV binds the low-rank Value representation to the output projection, so that the same set of low-rank factors both compresses the width of the Value cache and reduces tensor-parallel communication volume.
BasisKV reuses the compressed Value cache to predict part of the information needed for Key routing, and only reads back the original Keys for the selected Key pages to compute exact QK scores.
BasisKV is evaluated across multiple LLM families and system settings, and is reported to achieve a better quality-efficiency trade-off than existing low-rank baselines.
BasisKV achieves up to 1.42x end-to-end speedup under 8-GPU tensor parallel inference.
The BasisKV code has been open-sourced on GitHub at the repository alexz949/BasisKV.
All Facts (5)
BasisKV binds the low-rank Value representation to the output projection, so that the same set of low-rank factors both compresses the width of the Value cache and reduces tensor-parallel communication volume.
50%UnverifiedBasisKV reuses the compressed Value cache to predict part of the information needed for Key routing, and only reads back the original Keys for the selected Key pages to compute exact QK scores.
50%UnverifiedBasisKV is evaluated across multiple LLM families and system settings, and is reported to achieve a better quality-efficiency trade-off than existing low-rank baselines.
50%UnverifiedBasisKV achieves up to 1.42x end-to-end speedup under 8-GPU tensor parallel inference.
50%UnverifiedThe BasisKV code has been open-sourced on GitHub at the repository alexz949/BasisKV.
50%