Unverified50% confidenceBenchmarkExact time
The bfloat16 baseline throughput is 18.8K tokens/s/GPU, while the fastest custom path reaches 37.9K tokens/s/GPU, nearly doubling the throughput.
1
Sources
50%
Confidence
Long-term
Relevance
10/4/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedWhen the context of LLaMA-2 7B is extended to 32K tokens, KV Cache GPU memory usage expands to approximately 16 GB, approaching the total memory of a single consumer-grade GPU.72% similarUnverifiedCompressing the context from 128K to 32K may quadruple the number of concurrent users a single GPU can serve, representing a trade-off between throughput and context length.70% similarUnverifiedOn a single 32GB GPU under q4_0 quantization, the context length increased from 82,432 tokens before the update to 199,680 tokens after the update.68% similarVerifiedA single sequence of 32K context length for LLaMA-2 70B can require approximately 80 GB of KV cache in FP16.68% similarUnverifiedWith an approximately 1.6M ultra-long context and 128 concurrent requests, throughput can reach approximately 3,700 tokens/s (aggregated).68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/971841API
curl https://kongchang.com/api/v1/knowledge/claims/971841MCP
get_claim(id=971841)