KV Cache

When generating each token, a Transformer model needs to compute the Key and Value vectors of all previous tokens, and these tensors are collectively called the KV cache.

TTFT (time to first token) is mainly determined by the prefill stage, with computation roughly proportional to context length; prefix caching can reuse already-computed KV states and only prefill the newly added portion.

Model weights are only part of VRAM usage; during inference, space must also be reserved for activations, KV cache, context window, and so on.

Running the same Q8_0 file twice, the second time with q8_0 K/V cache, gives KL 0.0038, a larger effect than quantizing weights from BF16 to Q8_0 (KL 0.0022).

Because expert pruning preserves the attention architecture and KV cache format, the KV cache handoff requires no cache conversion, and the runtime only needs to switch weight masks between the two stages.

For multi-turn conversations or long-document question answering, if the KV of the entire context is recomputed on every request, the compute overhead grows quadratically with context length.

The order in which prompts are organized directly affects the cache hit rate, and thus the actual bill.

Actual token generation speed usually reaches only 50%–70% of the theoretical value, affected by factors such as compute unit scheduling, KV cache reads and writes, and quantization-dequantization overhead.

The prefill phase processes the complete input context, computes attention for all input tokens, and generates the KV cache; it is a compute-intensive operation whose time consumption is strongly correlated with input length.

vLLM implements a bidirectional KV transfer mechanism in which decode nodes send computed KV blocks back to prefill nodes, eliminating repeated prefill computation of common prefixes in multi-turn conversations.

In an Agent session, multiple rounds of calls share the same context prefix, so in theory there are a large number of reusable KV cache blocks.