Latent Compaction
A method that compresses the KV cache by treating it as a trainable neural network through test-time training. It learns a compact cache for each document while keeping model weights frozen, and it improves long-context inference efficiency without access to downstream question-answering data.
Timeline (last 90 days)
Latent Compaction treats the KV cache itself as a trainable neural network and optimizes it through test-time training.
Latent Compaction was proposed by the THUNLP team at Tsinghua University.
Latent Compaction achieves 96.7% accuracy on the RULER benchmark at a 32x compression rate, while KVZip+ achieves only 40.0%.
Latent Compaction training uses only generic prompts about the document and does not access any downstream questions or answers.
Latent Compaction learns a compact, document-specific cache for each document, while the model weights remain frozen.
Latent Compaction's cache entry initialization fuses three signals: reconstruction importance, teacher attention, and key-space diversity.
Latent Compaction training is divided into two stages: the first stage fits the attention output and log normalizer of the full cache, and the second stage jointly optimizes the entire cache through end-to-end distillation.
Latent Compaction achieves retrieval gains on both Qwen3 and MiniCPM5, demonstrating cross-model-family transferability.
Latent Compaction demonstrates advantages on benchmarks such as RULER, LongBench, and Phonebook Lookup.
The Latent Compaction code has been open-sourced on GitHub, with the repository THUNLP/latent-compaction.
1 more timeline events
All Facts (11)
The trend revealed by Latent Compaction is that static components in the inference process can dynamically adapt to the input through test-time training.
50%UnverifiedLatent Compaction was proposed by the THUNLP team at Tsinghua University.
50%UnverifiedLatent Compaction achieves 96.7% accuracy on the RULER benchmark at a 32x compression rate, while KVZip+ achieves only 40.0%.
50%UnverifiedLatent Compaction training uses only generic prompts about the document and does not access any downstream questions or answers.
50%UnverifiedLatent Compaction learns a compact, document-specific cache for each document, while the model weights remain frozen.
50%UnverifiedLatent Compaction's cache entry initialization fuses three signals: reconstruction importance, teacher attention, and key-space diversity.
50%UnverifiedLatent Compaction training is divided into two stages: the first stage fits the attention output and log normalizer of the full cache, and the second stage jointly optimizes the entire cache through end-to-end distillation.
50%UnverifiedLatent Compaction achieves retrieval gains on both Qwen3 and MiniCPM5, demonstrating cross-model-family transferability.
50%UnverifiedLatent Compaction demonstrates advantages on benchmarks such as RULER, LongBench, and Phonebook Lookup.
50%UnverifiedLatent Compaction treats the KV cache itself as a trainable neural network and optimizes it through test-time training.
50%UnverifiedThe Latent Compaction code has been open-sourced on GitHub, with the repository THUNLP/latent-compaction.
50%