Unverified50% confidenceBenchmarkExact time
SGLang using official NVFP4 weights with D-Spark speculative decoding achieved an average of approximately 173 tokens/second on Blackwell GPUs, approaching 200 for short tasks, while loading a 262k context window.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
9/16/2026
First Seen
Valid until: 12/15/2026
Sources
Related Entities
Related Claims
UnverifiedTaking Llama-3 70B as an example, the prefill stage for processing 32K tokens on an A100 80GB GPU takes approximately 800–1200ms, while processing 2K tokens takes only about 50ms.72% similarUnverifiedIn the STM32 embedded development test, the locally deployed DS4 model had an average output speed of approximately 23 tokens/second, memory usage spiked to 110 GB during inference, and the peak speed reached 83.8 tokens/second.70% similarUnverifiedThe KV cache for a 128K context window of a GPT-4-level model can occupy tens of gigabytes of VRAM70% similarUnverifiedThe core bottleneck in large model inference on GPUs is memory bandwidth; autoregressive generation of each token requires loading the full KV Cache from HBM, with compute utilization typically below 5%.69% similarUnverifiedOn an A100 (80 GB VRAM) GPU, the throughput for training a 7B model is approximately 3,000–5,000 tokens per second.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/924475API
curl https://kongchang.com/api/v1/knowledge/claims/924475MCP
get_claim(id=924475)