Unverified50% confidenceSolutionExact time
Counting throughput by counting streaming chunks will be severely underestimated (43 tok/s measured), because under speculative decoding vLLM sends the entire accepted batch (about 2.5 tokens per chunk); the correct approach is to read usage.completion_tokens from the final chunk and enable stream_options: include_usage.
1
Sources
50%
Confidence
Long-term
Relevance
10/16/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedA 27B-parameter Q8_0 model, together with vLLM's KV cache and runtime overhead, requires nearly 50 GB of memory space for a single inference service.67% similarUnverifiedWhen concurrent requests reach 150, the overall system throughput of the new vLLM version can increase to 5.3 times the original.66% similarUnverifiedPagedAttention causes KV cache memory waste to occur only in the last unfilled block of a request, and vLLM reports waste of less than 4%.65% similarUnverifiedPrefix caching can save 50%–80% of prefill-stage computation by reusing already-computed KV Cache, and vLLM implements fine-grained cache hit matching through a radix tree structure.64% similarUnverifiedIn the replaced model, the tied embedding/lm_head matrix remains bf16, so each token actually reads about 1.263 GB of weights (plus about 0.06 GB of KV cache); calculated by actual bytes moved, 25.5 tok/s corresponds to about 34 GB/s, roughly 10% of measured bandwidth.63% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1043377API
curl https://kongchang.com/api/v1/knowledge/claims/1043377MCP
get_claim(id=1043377)