KV-cache
KV cache is a technique used during Transformer model inference to store key-value pairs in order to accelerate autoregressive generation, reducing redundant computation.
Core Facts
The longer the context, the more the KV cache grows linearly, making memory bandwidth and capacity the main bottleneck for inference throughput.
KVCache is a mechanism in the Transformer architecture that caches the Key and Value vectors of all previous tokens to avoid redundant computation, with its memory footprint growing linearly with context length.
During autoregressive decoding, each newly generated token requires reading the full KV Cache, making the decoding stage almost always memory-bandwidth-bound rather than compute-bound.
Timeline (last 90 days)
KV-Surgeon is an offline KV-cache compression method that uses calibration data to account for interactions among cache entries, deriving a quadratic surrogate model to make joint greedy deletion decisions and performing compensation among the remaining entries.
KV-cache reuse presupposes a stable prefix, and GraphRAG flattens the graph structure into linear text, so the context prefix differs almost every query and the cache hit rate is extremely low.
KV-cache size is proportional to the number of sequence tokens and directly determines the number of concurrent requests a single card can handle; more tokens mean a larger KV-cache per user, and the number of concurrent users on the same GPU shrinks proportionally.
HSQ-A uses a lightweight scoring network to select about 15% of high-impact keys and stores the remaining values in FP8 format, thereby compressing the KV-cache from the 0.8 GB/k common in dense 70B models to 0.12 GB/k, a compression ratio of about 85%.
Implementing instant cancellation of LLM streaming responses requires an unbroken signal propagation chain: the client's AbortController.abort() triggers HTTP fetch teardown, the gateway detects the socket close via the server context and cancels the upstream request context, and the model inference runtime catches the disconnection, stops autoregressive generation, and releases the KV-cache.
The longer the context, the more the KV cache grows linearly, making memory bandwidth and capacity the main bottleneck for inference throughput.
KVCache is a mechanism in the Transformer architecture that caches the Key and Value vectors of all previous tokens to avoid redundant computation, with its memory footprint growing linearly with context length.
Normalizing flows share with large language models the same causal mask, KV-cache mechanism, and a consistent left-to-right sequence generation structure.
During autoregressive decoding, each newly generated token requires reading the full KV Cache, making the decoding stage almost always memory-bandwidth-bound rather than compute-bound.
All Facts (9)
The longer the context, the more the KV cache grows linearly, making memory bandwidth and capacity the main bottleneck for inference throughput.
75%VerifiedKVCache is a mechanism in the Transformer architecture that caches the Key and Value vectors of all previous tokens to avoid redundant computation, with its memory footprint growing linearly with context length.
75%VerifiedDuring autoregressive decoding, each newly generated token requires reading the full KV Cache, making the decoding stage almost always memory-bandwidth-bound rather than compute-bound.
75%UnverifiedKV-Surgeon is an offline KV-cache compression method that uses calibration data to account for interactions among cache entries, deriving a quadratic surrogate model to make joint greedy deletion decisions and performing compensation among the remaining entries.
50%UnverifiedKV-cache reuse presupposes a stable prefix, and GraphRAG flattens the graph structure into linear text, so the context prefix differs almost every query and the cache hit rate is extremely low.
50%UnverifiedKV-cache size is proportional to the number of sequence tokens and directly determines the number of concurrent requests a single card can handle; more tokens mean a larger KV-cache per user, and the number of concurrent users on the same GPU shrinks proportionally.
50%UnverifiedHSQ-A uses a lightweight scoring network to select about 15% of high-impact keys and stores the remaining values in FP8 format, thereby compressing the KV-cache from the 0.8 GB/k common in dense 70B models to 0.12 GB/k, a compression ratio of about 85%.
50%UnverifiedImplementing instant cancellation of LLM streaming responses requires an unbroken signal propagation chain: the client's AbortController.abort() triggers HTTP fetch teardown, the gateway detects the socket close via the server context and cancels the upstream request context, and the model inference runtime catches the disconnection, stops autoregressive generation, and releases the KV-cache.
50%UnverifiedNormalizing flows share with large language models the same causal mask, KV-cache mechanism, and a consistent left-to-right sequence generation structure.
50%