Unverified50% confidenceFactExact time
Actual token generation speed usually reaches only 50%–70% of the theoretical value, affected by factors such as compute unit scheduling, KV cache reads and writes, and quantization-dequantization overhead.
1
Sources
50%
Confidence
Long-term
Relevance
10/14/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe output speed of 94 tokens per second was measured at a specific quantization precision and context length, and the actual experience will fluctuate with factors such as prompt length and concurrency.75% similarUnverifiedAfter offloading half of the model layers to system memory, the generation speed often drops sharply from over ten tokens per second to single digits.72% similarVerifiedThe price of output tokens is typically 4–6 times that of input tokens, stemming from the computational differences between the prefill and decode phases, with GPU utilization during the decode phase potentially being only 10–20% of that during prefill70% similarUnverified传统推理框架中每个请求的 KV Cache 需预分配等长连续显存，显存利用率通常仅有 20%-40%，PagedAttention 可将其提升到接近 100%68% similarUnverifiedThe inference speed of a 70B model may be only one-tenth that of a 7B model, while its memory footprint is several times larger.68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1025808API
curl https://kongchang.com/api/v1/knowledge/claims/1025808MCP
get_claim(id=1025808)