Unverified50% confidenceBenchmarkExact time
On NVIDIA L4, the bf16 model (6.32 GiB) has a theoretical memory-bandwidth ceiling of about 47 tokens/s, while the int4 model (2.85 GiB) has a theoretical ceiling of about 105 tokens/s, with measured peak reaching 131.9 tokens/s.
1
Sources
50%
Confidence
Long-term
Relevance
10/12/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedIf bandwidth reached 2.4 TB/s, the theoretical speed ceiling for a 100 GB model could reach 24 tokens/s, a 60 GB model 40 tokens/s, and a 35 GB model close to 69 tokens/s.76% similarUnverifiedFor a typical 7B model (32 layers, 32 heads, 128 dimensions) processing a 2048-token sequence at FP16 precision with a batch size of 8, the KV Cache consumes approximately 4 GB of VRAM.71% similarVerified以LLaMA-3 70B（80层、64头、每头128维）为例，单个token的KV Cache约占2.6MB（FP16精度），32K上下文的KV Cache总量约达83GB70% similarUnverifiedTaking a 70B BF16 model as an example, generating one token requires reading approximately 140 GB of weights, and when fully utilizing the H100's 3.35 TB/s memory bandwidth, the per-token latency is at least 42 milliseconds, with a theoretical TPS upper limit of approximately 24 tokens/second.70% similarVerifiedA 70B-parameter FP16 model requires approximately 140 GB of video memory, approximately 70 GB after INT8 quantization, and approximately 35–40 GB after INT4 quantization.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1011105API
curl https://kongchang.com/api/v1/knowledge/claims/1011105MCP
get_claim(id=1011105)