Unverified50% confidenceFactExact time
The roughly 14% gap between the theoretical prediction of 36 tokens/second and the measured 31 tokens/second mainly comes from memory reads not reaching peak bandwidth, the computational overhead of the attention mechanism, and scheduling losses in the software stack.
1
Sources
50%
Confidence
Long-term
Relevance
10/20/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedA context length of 131,072, i.e., 128K tokens, leads to significantly higher VRAM consumption and latency during inference as the context grows longer.78% similarUnverifiedEmpirically, above 20 token/s is close to reading speed and allows smooth interaction, while below 5 token/s is noticeably laggy.75% similarUnverifiedBecause thinking tokens count toward max_tokens, a low max_tokens budget set for a non-thinking model will squeeze the answer space, causing responses to be too short or truncated; it is recommended to keep it ample (e.g., 32,000), and for long agent runs raise it to 256K or above per the model card's recommendation.72% similarUnverifiedIn actual testing, the prefill read speed for a 32K context reached 3,860 tokens per second.71% similarUnverifiedThe baseline BF16 model reaches about 37.5 tokens/s at single concurrency, 714 tokens/s aggregate throughput at 32 concurrent requests, and 61ms first-token latency; after switching to FP8 weights, single concurrency rises to 53.4 tokens/s, and peak throughput at 64 concurrency rises from about 800 to 1,140 tokens/s, about 1.42x.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1075812API
curl https://kongchang.com/api/v1/knowledge/claims/1075812MCP
get_claim(id=1075812)