Unverified50% confidenceFactExact time
Estimating a single-token KV Cache at about 1 MB, moving the cache for 8K tokens over PCIe takes several hundred milliseconds, enough to double the first-token latency.
1
Sources
50%
Confidence
Long-term
Relevance
10/19/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedFor a typical 7B-parameter model, the KV Cache per 1K tokens requires approximately 0.5–1 GB of VRAM.78% similarUnverifiedDuring the prefill phase, Strata splits the prompt into chunks of up to 8,000 tokens and streams them over PCIe hidden behind computation, boosting the speed of processing a 32,000-token prompt from approximately 570 tokens/second to nearly 1,300 tokens/second.75% similarUnverifiedWhen processing a long document of 8000 tokens, the KV cache may consume several additional GB of memory.74% similarUnverifiedWhen implementing a KV cache, one should compare the cached and uncached versions token by token on greedy output over long sequences (e.g., more than 200 tokens) rather than only performing single-step validation, in order to effectively verify cache correctness.74% similarUnverified上下文越长，模型需要缓存的 KV Cache 越大，KV Cache 使模型生成新 Token 时无需重新计算所有历史 Token 的注意力权重74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1069703API
curl https://kongchang.com/api/v1/knowledge/claims/1069703MCP
get_claim(id=1069703)