Unverified50% confidenceBenchmarkExact time
Long context has little effect on generation speed: after 4K context the median time per output token is 7.6 ms, and after 256K it is 8.3 ms; but the first read of a 128K-token prompt takes 18.33 seconds, so prefix caching is critical.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/16/2026
First Seen
Valid until: 1/14/2027
Sources
Related Entities
Related Claims
UnverifiedOn an 8GB GPU, the token generation speed is approximately 4.2 to 5 tokens/second, and decreases from around 6 tokens/second to approximately 4.2 as the context grows longer.76% similarUnverifiedAt approximately 4K input, TTFT is 23.6 seconds with decoding at 46.6 tok/s; at 8K, 47 seconds and 41.4 tok/s; at 16K, 1 minute 30 seconds and 30.8 tok/s; at 30K, 3 minutes 15 seconds and 31.9 tok/s.75% similarUnverifiedIt is generally considered that single-concurrency output above 100 tokens/second is sufficient to achieve a smooth typewriter effect, and at approximately 200 tokens/second, generating a 2,000-character long text takes only about 10 seconds.74% similarVerifiedIn decode speed tests, speeds of approximately 45 tokens/second are achievable within 20,000 tokens, dropping sharply to approximately 20 tokens/second beyond 30,000 tokens.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1044823API
curl https://kongchang.com/api/v1/knowledge/claims/1044823MCP
get_claim(id=1044823)