SGLang Cold Start Measured: 88% of the T4 Bill Is Spent Before the First Request

One T4 GPU, one lightweight model, one cold start — and 88% of the bill is burned before the first user gets an answer. This isn't hyperbole for a catchy headline; it's measured data with a raw timeline attached.

Experimental Setup: Simulating Scale-to-Zero Billing

The test environment was deliberately modest: a single T4 on Colab, SGLang 0.5.21, running the fp16 version of Qwen2.5-0.5B-Instruct. The more noteworthy choice is the author's methodology — timing starts before the server launches, simulating how a scale-to-zero platform begins billing the moment the worker is created.

The flow is: start the service, send 8 short chat requests, run a 60-second health check, then stop. One run, fully recorded.

The key to this design is that it doesn't start counting from "request arrival" but from "resource allocation." That's exactly how on-demand GPU platforms actually bill, and it's the starting point many developers overlook when estimating costs.

scale-to-zero is a resource scheduling strategy used by cloud-native and on-demand GPU platforms: when a service receives no requests for some period, the platform fully destroys the backend worker instance so it no longer occupies the GPU; when a new request arrives, it spins up a fresh instance. Its biggest benefit is that idle time costs nothing, which is why it's widely adopted by serverless inference platforms. The trade-off is that every cold start goes through the full process initialization, model loading, memory allocation, and graph capture sequence — during which users can only wait, and the platform typically starts billing the moment the instance begins creation. The key to understanding scale-to-zero is distinguishing three kinds of time: platform billing time, the process's self-reported ready time, and the time the user actually gets a response. These three often diverge dramatically, and estimating costs using only the middle "ready" value will severely underestimate the real bill.

The Timeline: What Does "Ready" Actually Mean?

The raw timeline is the most valuable part of this article:

• 0s: Server starts, billing begins
• ~371s: Prefill CUDA graph capture completes, taking 287.7s
• 386.7s: /health returns 200, logs show "ready to roll"
• ~563s: First chat request completes, taking ~176s
• ~579s: Second request completes (~16s), the remaining six finish within the same second
• 639.6s: Billing stops

After that, sending another 8 requests on the same server takes only 0.08 to 0.22 seconds each.

If you slice the bill at the health check point, startup looks like it accounts for 60% (386.7s out of 639.6s). But the first user didn't get an answer until ~563s — by which point 88% of the bill was already spent.

"Ready" isn't the end of the wait — it's two-thirds of the wait.

Why CUDA Graph Capture Becomes a Hidden Cost

The 287.7-second prefill CUDA graph capture is the single largest block in the entire timeline. CUDA graph capture essentially records a series of GPU operations so they can be quickly replayed later, reducing CPU scheduling overhead. For inference services this is a performance optimization, but in a cold-start scenario it becomes pure warm-up cost.

There's an easy-to-misunderstand point here: the logs don't say what the 176 seconds of the first request were actually spent on, so the author explicitly refrains from causal claims. That restraint deserves respect — separating observation from speculation is basic discipline for measurement-focused content.

Another detail is subtler: each trailing health check shows up in SGLang logs as a one-token prefill on the GPU, yet the request counter doesn't move. This means "the GPU did work" and "the request was served" are two different things. The health check itself consumes compute but produces no user value and doesn't appear in request metrics.

CUDA graph is an execution optimization mechanism NVIDIA introduced in CUDA 10. Normally, every kernel launch and memory copy on the GPU is submitted one by one by the CPU; in inference workloads with many kernels and short per-kernel durations, CPU submission overhead can exceed actual GPU compute time, creating a bottleneck. CUDA graph lets you pre-record an entire chain of GPU operations (including dependencies) as a directed acyclic graph, then replay the whole thing with a single submission, drastically cutting CPU scheduling overhead and launch latency. The cost is that the recording phase itself requires running the graph through once or even multiple times, and support for dynamic shapes and dynamic control flow is limited. In long-running services, the recording cost gets amortized across thousands of inferences and becomes nearly negligible; but in a scale-to-zero cold start, the recording happens only once and must be put to use immediately, so it turns from a "performance optimization" into a real "warm-up tax." The prefill stage in particular, because it must handle variable-length inputs, often requires capturing separately for many sequence lengths, further inflating recording time.

Side-by-Side Comparison with vLLM

The author offers a reference point: on the same card, vLLM 0.27.1 took only 5.8 seconds total for 8 requests after ready. vLLM's startup overhead was about 165 of 171 billed seconds.

This comparison needs careful interpretation. The two frameworks differ in versions, optimization strategies, and default configurations, and a single run can't represent all scenarios. But it at least shows one thing: the composition of cold start varies significantly between inference frameworks, and post-ready behavior does too.

The author himself emphasizes that this is a single-card, single run, and that a larger GPU or a different model could be completely different. That's not a disclaimer — it's the core point: measure your own scenario.

vLLM is one of the most mainstream open-source large-model inference engines today, known for PagedAttention memory management and high throughput, standing alongside SGLang as one of the two major high-performance inference frameworks. Both adopt ideas like paged KV cache and continuous batching at the architectural level, but their default configurations, graph capture strategies, and scheduler implementations differ substantially. vLLM typically only does CUDA graph capture for the decode stage at startup, and applies more aggressive trimming to the set of captured batch sizes, which tends to make its cold-start graph capture phase shorter than SGLang's. At the same time, default switches like prefix caching and chunked prefill also affect first-request latency. So the comparison in the article — "same T4, vLLM takes only 5.8s for 8 requests after ready" — reflects not just a throughput gap, but more deeply a different trade-off between the two frameworks on how much warm-up work to do at startup: one leans toward doing more preparation at startup so it's faster after ready, the other leans toward becoming ready ASAP and shifting some of the cost onto the first request.

Three Timestamps for Scale-to-Zero Users

Based on this measurement, the author recommends recording three timestamps when running a scale-to-zero service, not just one:

1. Billing start to health check passing
2. Billing start to first completed request
3. The idle tail after the last request

Then price by the entire billing duration, rather than only the portion where requests were actually running.

The practical significance of this method is that many teams estimate GPU costs using "request processing time," ignoring the overhead of cold start, warm-up, and idle tail. When a service scales up and down frequently, this portion of cost can become the bulk of the bill.

The author has put the method and the open-source tools used on GitHub (github.com/qaisermehdi3-coder/qvunex), and is openly soliciting measurement data from other card types and servers. This openness makes it possible for a single-point measurement to become a community benchmark.

One-line Summary

SGLang cold start on T4, measured: 88% of billed time is consumed before the first user gets an answer — ready is not the same as available.

Paragraph Summary

Based on a measurement on a single T4 on Colab, SGLang 0.5.21 running Qwen2.5-0.5B, simulating a scale-to-zero platform that bills from worker creation. The raw timeline shows: health check passes at 386.7s, but the first user request completes only at 563s — 88% of the 639.6s billed duration. Of that, prefill CUDA graph capture takes 287.7s, the largest hidden cost; health checks produce one-token prefill on the GPU but don't count toward request metrics. As a reference, vLLM needs only 5.8s for 8 requests after ready in the same environment. The author accordingly recommends pricing by the full billed duration rather than request runtime, recording three timestamps, and emphasizes that a single run can't represent all scenarios — measure your own actual environment.