Unverified50% confidenceBenchmarkExact time
vLLM's bidirectional KV transfer reduces TTFT in a 24k token scenario on Qwen3-32B from 778ms to 460ms, about a 1.6x improvement.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/14/2026
First Seen
Valid until: 1/12/2027
Sources
Related Entities
Related Claims
UnverifiedThrough request-owned speculative KV, in a DeepSeek-V4-Pro 1P1D, MI355X, concurrency-256 scenario, the average TTFT was reduced from 19.4 seconds to 15.1 seconds.74% similarUnverifiedThe Q4_K_M/GGUF version of Qwen 27B decodes at approximately 10.7 tok/s on the same machine, roughly 1/3.8 of the MLX version.70% similarUnverifiedThe context length of mainstream open-source multimodal models is typically between 4K and 32K tokens: LLaVA-1.5 is 4,096 tokens, Qwen-VL is 8,192 tokens, and InternVL2 extends to 32K.70% similarUnverifiedOn an RTX 3060, the developer's hand-written Triton fused W4A16 GEMV kernel targeting batch-1 decoding runs at about 25.5 tok/s, matching the roughly 25.7 tok/s of the bitsandbytes NF4 quantization scheme.69% similarUnverifiedOne B70 user running vLLM + GPT-Q INT4 quantization with MTP depth 4 achieved 84.6 tokens/s on a 27B model, but this only supports single-stream and uses different quantization, making direct comparison impossible.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1026256API
curl https://kongchang.com/api/v1/knowledge/claims/1026256MCP
get_claim(id=1026256)