Unverified50% confidenceBenchmarkExact time
On the T4, --fit plus a 10% cache reached 23.6 tokens/s with a 75.6% hit rate; --cpu-moe alone was 9.6 tokens/s; --cpu-moe plus a 50 MiB cache was only 3.8 tokens/s.
1
Sources
50%
Confidence
Short-term (~14 days)
Relevance
10/18/2026
First Seen
Valid until: 11/1/2026
Sources
Related Entities
Related Claims
UnverifiedThe author considers 12–30 tokens/s to be the comfort zone, with above 30 tokens/s being good; an M5 Max with 48–64 GB can run at approximately 20 tokens/s.76% similarUnverifiedEmpirically, above 20 token/s is close to reading speed and allows smooth interaction, while below 5 token/s is noticeably laggy.73% similarUnverifiedDeveloper Primitive reported: 91.2 tokens/s with speculative decoding off, reaching 142.6 tokens/s after enabling 3 speculative tokens, approximately a 56% speedup under the same configuration.73% similarUnverifiedUnder high power operation, the 35B MoE model reaches 78 to 107 tokens/s, while the dense 27B model only reaches 22 tokens/s.72% similarUnverifiedConcurrent requests amplify the demand for cache slots, causing more frequent LRU evictions; on the T4, four concurrent requests achieved total throughput of 37.0 tokens/s, lower than the 43.2 tokens/s of a single request, with the hit rate dropping from 95.7% to 92.3%.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1060511API
curl https://kongchang.com/api/v1/knowledge/claims/1060511MCP
get_claim(id=1060511)