Unverified50% confidenceBenchmarkExact time
On a single NVIDIA A40 with fp32 and TF32 disabled, batch size 1 and greedy decoding, after GPT-2 small uses the KV cache, single-token latency at context length 1000 drops from 29.69 ms to 4.95 ms, about 6x.
1
Sources
50%
Confidence
Long-term
Relevance
10/18/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe T4 GPU has 16GB of VRAM and 8.1 TFLOPS of FP32 compute performance.71% similarUnverifiedIn the test, model loading took only a few seconds, GPU usage dropped from 23GB to about 17GB, the first test prompt processing took 225ms, generation speed was stable at 33.5 tok/s, with a total output of 1500 tokens.71% similarUnverifiedThe RTX 4090 has a theoretical FP16 compute performance of 330 TFLOPS, and the actual Decode speed for a 7B FP16 model is typically 50–60 tokens/s.70% similarUnverifiedLFM2.5-24B-A2B achieves throughput of over 21,000 tokens/second on a single NVIDIA B300 private cloud accelerator.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1060139API
curl https://kongchang.com/api/v1/knowledge/claims/1060139MCP
get_claim(id=1060139)