Unverified50% confidenceFactExact time
Adding weights of 39.8 GiB, KV Cache of 39.1 GiB, and activations and runtime of about 4.2 GiB gives a total of about 83.1 GiB, which will cause OOM on an 80GB GPU.
1
Sources
50%
Confidence
Long-term
Relevance
10/14/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedOn NVIDIA L4, the bf16 model (6.32 GiB) has a theoretical memory-bandwidth ceiling of about 47 tokens/s, while the int4 model (2.85 GiB) has a theoretical ceiling of about 105 tokens/s, with measured peak reaching 131.9 tokens/s.69% similarUnverifiedIn an experimental configuration, Ollama identifies the 840M as having approximately 16 GB of VRAM (16193 MiB), with about 14.6 GiB available for the Vulkan device at load time.69% similarUnverifiedRunning a 16-bit precision 70B model on 2×80GB GPUs leaves only 2.9 GB of VRAM after the weights are loaded, accommodating only about 9,000 in-flight tokens, which in practice cannot serve requests effectively.69% similarVerifiedTaking a 70B parameter model with FP16 precision and 32 attention head layers as an example, the KV Cache for a single 200K-length request requires approximately 80–120 GB of VRAM, far exceeding the 80 GB VRAM limit of a single A100 GPU.69% similarUnverifiedA 70.6B model using the GGUF Q4_K_M format has a weight size of about 39.8 GiB.68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1025445API
curl https://kongchang.com/api/v1/knowledge/claims/1025445MCP
get_claim(id=1025445)