T4
The NVIDIA Tesla T4 is a cloud inference GPU based on the Turing architecture. It uses passive cooling and is commonly deployed in data centers and cloud services.
Core Facts
Timeline (last 90 days)
The author fixed the thresholds before running the evaluation, among which metric T4 requires that the proportion of "incorrectly judged as covered" not exceed 5%.
The author reproduced it on a T4 with the same model and testing method: at 8 concurrency, a single request decodes at 122–124 tokens/s, and the whole server produces 701–708 tokens/s, likewise about a 5.7-fold difference.
Running SGLang 0.5.21 on a single T4 in Colab to load the Qwen2.5-0.5B-Instruct fp16 model, in one complete cold-start measurement, 88% of the billed time was spent before the first user request completed.
On the same T4, vLLM 0.27.1 took only 5.8 seconds in total for 8 requests after ready; vLLM's startup overhead was about 165 of 171 billed seconds.
On a T4 GPU, Laya takes 33 milliseconds per single question, and batch processing can compress that to 7 milliseconds
NVIDIA T4是2018年发布的推理优化GPU，拥有65 TFLOPS的FP16算力和16GB显存
作者称CWAA V6在一块T4 GPU上可达到0.38秒/步的训练速度且高度稳定
All Facts (7)
On a T4 GPU, Laya takes 33 milliseconds per single question, and batch processing can compress that to 7 milliseconds
65%UnverifiedThe author fixed the thresholds before running the evaluation, among which metric T4 requires that the proportion of "incorrectly judged as covered" not exceed 5%.
50%UnverifiedThe author reproduced it on a T4 with the same model and testing method: at 8 concurrency, a single request decodes at 122–124 tokens/s, and the whole server produces 701–708 tokens/s, likewise about a 5.7-fold difference.
50%UnverifiedRunning SGLang 0.5.21 on a single T4 in Colab to load the Qwen2.5-0.5B-Instruct fp16 model, in one complete cold-start measurement, 88% of the billed time was spent before the first user request completed.
50%UnverifiedOn the same T4, vLLM 0.27.1 took only 5.8 seconds in total for 8 requests after ready; vLLM's startup overhead was about 165 of 171 billed seconds.
50%Unverified作者称CWAA V6在一块T4 GPU上可达到0.38秒/步的训练速度且高度稳定
50%UnverifiedNVIDIA T4是2018年发布的推理优化GPU，拥有65 TFLOPS的FP16算力和16GB显存
50%