Unverified50% confidenceBenchmarkExact time
In single-user scenarios llama.cpp runs an 8B model at about 115 tok/s with about 12ms first-token latency on an RTX 4090, but at 32 concurrency throughput is only about 210 tok/s.
1
Sources
50%
Confidence
Short-term (~14 days)
Relevance
10/19/2026
First Seen
Valid until: 11/2/2026
Sources
Related Entities
Related Claims
UnverifiedThe RTX 4090 has a theoretical FP16 compute performance of 330 TFLOPS, and the actual Decode speed for a 7B FP16 model is typically 50–60 tokens/s.73% similarUnverifiedWith a 5070 12 GB GPU paired with 64 GB of memory, speed can reach 93 tokens/s in short chat scenarios and 74 tokens/s for 128K long contexts.72% similarUnverifiedIn terms of prompt processing speed, the RTX 4090 achieves approximately 3,000 tokens/s and the M5 Max approximately 800 tokens/s, giving NVIDIA a clear advantage in long-document and codebase input scenarios.72% similarUnverifiedA UP host running MiniCPM 2B with an RTX 4080S (16GB VRAM) + 64GB RAM configuration achieved an inference speed of approximately 110 tokens/second in real-world testing71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1067962API
curl https://kongchang.com/api/v1/knowledge/claims/1067962MCP
get_claim(id=1067962)