Unverified50% confidenceFactExact time
An 8192-token prompt on a 70B model produces 2.68 GB of KV cache in BF16.
1
Sources
50%
Confidence
Long-term
Relevance
10/19/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedTaking LLaMA-2 70B as an example, when processing a 4096-token context with a batch size of 32, the KV Cache can consume over 80 GB of memory.79% similarVerifiedTaking the LLaMA-2 70B model as an example, a sequence of 2048 tokens requires approximately 4 GB of VRAM for the KV cache alone.78% similarVerifiedFor a model with 70B parameters, the KV Cache for an 8K context may occupy several additional gigabytes of VRAM76% similarVerifiedLLaMA-2-70B's KV Cache for a single request at a context length of 128K can occupy more than 160 GB of VRAM.75% similarUnverifiedThis experiment was configured with a native context of 262,144 tokens, the KV cache stored in FP8, and a maximum of two concurrent sequences.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1069155API
curl https://kongchang.com/api/v1/knowledge/claims/1069155MCP
get_claim(id=1069155)