DCKV: An End-to-End Learned KV Cache Codec with Over 100x Compression

Hybrid language models retain key-value (KV) caches only in a small number of full-attention layers, but this portion of the cache is precisely the part of the prompt state that grows with prompt length and needs to be stored or migrated for reuse. Most existing compression methods are optimized independently of the model, whereas DCKV proposes a differentiable KV cache codec that, for the first time, enables end-to-end training by jointly optimizing the transform, quantizer, and entropy model with a frozen language model.

Why KV Caches Need a Dedicated Codec

In hybrid architectures, the number of full-attention layers is limited, so the KV cache size is relatively manageable—but it remains the only part of the prompt state that grows linearly with prompt length. When prompt states need to be reused—for example, in multi-turn dialogue, retrieval-augmented generation, or long-document reasoning—this portion of the cache must be stored or migrated across devices. Traditional formats such as bf16 occupy a high number of bytes, making direct storage or transmission costs significant.

The core insight of DCKV is that the KV cache is not ordinary data: its value lies in enabling the language model to recover the correct output. Therefore, the compression objective should not merely minimize reconstruction error but should also account for the impact of the compressed cache on the model's output behavior. This approach is fundamentally different from traditional general-purpose compression algorithms.

Hybrid Language Model refers to models that alternate or combine full-attention layers with efficient layers such as linear attention or state space models (e.g., Mamba). Full-attention layers must store key-value vectors for every historical token, with overhead growing linearly with sequence length; linear attention layers instead replace the KV cache with a fixed-size recurrent state. Thus, hybrid architectures concentrate the KV cache in a few full-attention layers, making the overall size more manageable than in pure Transformers—but the retained portion of the cache is precisely the part that grows with prompt length and needs to be reused across sessions or devices. DCKV specifically compresses the caches of these full-attention layers, which is why it emphasizes being "designed for hybrid architectures."

End-to-End Training: Joint Optimization of Transform, Quantization, and Entropy Model

The DCKV codec comprises three learnable components: a transform, a quantizer, and an entropy model. The transform maps the KV cache into a more compact representation space; the quantizer discretizes continuous values; and the entropy model performs probabilistic modeling of the quantized symbols for entropy coding. The three are connected differentiably and optimized end-to-end together with a frozen language model.

The training objective consists of two terms: the code length assigned by the entropy model (i.e., the compression rate) and the change in model output (i.e., the distortion introduced by compression). Since the language model parameters are frozen, gradients are backpropagated through the model output to the codec, enabling it to learn to minimize the impact on model behavior at a given bitrate. This joint optimization allows DCKV to adaptively allocate bits for a specific model and task.

The conditional design of the entropy model is also noteworthy: it is conditioned on prompt tokens and previously decoded neighboring positions, leveraging contextual information to improve probability prediction accuracy. This means the decoding process is autoregressive, where later positions can depend on previously recovered content, thereby further reducing code length.

Entropy Coding and Entropy Models: Entropy coding is a class of lossless compression methods (such as arithmetic coding and rANS) that assigns code lengths based on the probability distribution of symbols—high-probability symbols get short codes and low-probability symbols get long codes, theoretically approaching the information entropy of the data. To use entropy coding, one must first have a model that can estimate symbol probabilities, i.e., an entropy model. In learned compression, the entropy model is typically a learnable neural network whose output probability distribution is used both for encoding and directly as the rate term (expected code length) in the training loss. DCKV's innovation is to condition the entropy model on prompt tokens and previously decoded neighboring positions, effectively using context to improve probability prediction accuracy—at the cost of requiring sequential autoregressive decoding.

Differentiable Quantization: Quantization maps continuous values to a discrete codebook and is inherently non-differentiable, so gradients cannot pass directly through the rounding operation. Common solutions include the straight-through estimator (rounding in the forward pass and passing gradients back unchanged in the backward pass), soft quantization, or additive uniform noise approximation. DCKV requires joint end-to-end optimization of the transform, quantizer, entropy model, and frozen language model, so it must employ such differentiable quantization techniques to allow distortion gradients from the model output to backpropagate to the quantizer parameters.

Experimental Results: The Trade-off Between Compression Rate and Accuracy Loss

On the Qwen3.5-2B model, DCKV demonstrates considerable compression performance. According to the paper, DCKV reduces the stored bytes per token to more than 55x less than bf16, while accuracy loss on three long-context benchmarks does not exceed 1.8 percentage points. When the compression rate is further increased beyond 100x, accuracy loss remains within 3.6 percentage points.

This result indicates that there is substantial redundancy in KV caches, and that learned transform coding can effectively exploit this redundancy. Compared with fixed-format storage, DCKV significantly reduces storage and transmission overhead with almost no loss of model capability. For applications that require frequent reuse of prompt states, this compression ratio translates into significant cost savings.

Learned transform coding refers to using a learnable transform (rather than a fixed transform such as DCT) to map signals into a representation space more amenable to compression, followed by quantization and entropy coding—a mainstream paradigm in neural image/video compression. Its effectiveness relies on a prerequisite: the data contains exploitable redundancy and sparsity under an appropriate transform. DCKV transfers this paradigm to KV caches and uses language model output rather than pixel reconstruction as the distortion measure, effectively changing the objective of "transform coding" from signal fidelity to task fidelity.

Potential Impact on Long-Context Applications

One of the cost bottlenecks in long-context inference is the memory footprint and migration overhead of KV caches. DCKV's end-to-end compression approach offers a new way to mitigate this problem: when prompt states need to be persisted or shared across devices, they can be compressed before storage and decompressed upon use, trading a small amount of accuracy for substantial resource savings.

However, DCKV is still at the research stage; its encoding and decoding process itself requires additional computation, and the autoregressive nature of entropy coding may introduce decoding latency. Practical deployment requires a trade-off between compression benefits and codec overhead. Furthermore, the method is designed for hybrid language models, and its applicability to pure-attention architectures remains to be verified.

Overall, DCKV introduces learned compression into KV cache management and aligns the codec with language model outputs, providing a reference paradigm for efficient storage and reuse of prompt states. As long-context model applications deepen, such dedicated codec techniques targeting model internal states deserve continued attention.

One-line Summary

DCKV is the first end-to-end trained KV cache codec, jointly optimized with a frozen language model.

Paragraph Summary

DCKV is a KV cache compression scheme for hybrid language models and the first end-to-end trained KV cache codec. In hybrid architectures, only a few full-attention layers retain KV caches, but this portion is the key part of the prompt state that grows linearly with length and needs to be stored or migrated. DCKV differentiably connects three learnable components—a transform, a quantizer, and an entropy model—and jointly optimizes them with a frozen language model. The training objective accounts for both compression code length and the distortion of model outputs caused by compression, and the entropy model performs autoregressive decoding conditioned on prompt tokens and previously decoded neighboring positions. On Qwen3.5-2B, compressing to more than 55x fewer bytes than bf16 incurs no more than 1.8 percentage points of accuracy loss, and beyond 100x, no more than 3.6 percentage points. The method offers a new approach for efficient storage and reuse of long-context prompt states, though codec overhead and the range of applicable architectures still require further validation.