Same H100, Why Does Cost Per Million Tokens Differ by 5.5x?

Title

Same H100, Why Does Cost Per Million Tokens Differ by 5.5x?

Meta Description

A developer benchmarked vLLM on an H100: at the same hourly rental, the cost per million tokens for single-request versus 8-concurrent inference differs by about 5.5x. The article breaks down common cost-estimation pitfalls, the three variables that determine real cost, and why single-request speed cannot be used to infer inference cost.

Article Content

The same H100, the same hour, the same rental price—yet the cost per million tokens from single-request runs can be nearly 5.5x higher than under batched concurrency. A developer measured these numbers with vLLM and open-sourced them.

A Real H100 Cost Comparison Experiment

The setup was clean: an H100 SXM 80GB on vast.ai at $2.689/hour, 20 vCPUs. The inference framework was the official vLLM 0.27.1 image, the model was Qwen2.5-0.5B-Instruct (fp16), and all prompts were unique to avoid cache interference. The data source was not client-side estimation, but vLLM's own /metrics counters, which matched the client records exactly.

The test had two load types, each run three times:

• Single-request serial: each request decoded at 648–649 tokens/s, the whole server produced 589–593 tokens/s, equivalent to about $1.26 per million output tokens.
• 8-request concurrency: each request dropped to 549–573 tokens/s, but the whole server produced 3,194–3,348 tokens/s, equivalent to about $0.22–0.23 per million output tokens.

Same card, same hour, same price—the results differ by about 5.5x.

vLLM and Continuous Batching

vLLM is currently the leading open-source LLM inference engine. Its core innovation is PagedAttention—managing the KV Cache in blocks, much like an operating system manages memory paging, thereby greatly reducing VRAM fragmentation and improving VRAM utilization. Its other key feature is continuous batching: requests do not have to wait for an entire batch to finish before being released together; instead, with each generated token, new requests are dynamically inserted into the batch and completed requests are removed. This is precisely the underlying mechanism by which the concurrent load in the article significantly boosts whole-machine throughput—when the GPU's matrix computation units are shared across sequences from multiple requests, utilization is far higher than when serving just one request. The /metrics mentioned in the article is vLLM's exposed Prometheus-format monitoring endpoint, whose counters record the actual number of prompt tokens processed and tokens generated, making it more reliable than client-side timestamp estimation.

What Common Cost Estimates Get Wrong

Many cost estimates work like this: take one request's inference speed and divide it into the hourly rental. By that logic, at 8 concurrency a single request at about 549–573 tokens/s works out to $1.30–1.36 per million tokens per hour.

But the real cost is $0.22–0.23. In every round, the server's overall output was 5.65–5.84x the single-request perspective.

In other words, using single-request speed to infer cost would overestimate by about 5.7x under this load. Under a different load, that multiplier changes again—the key point is that the multiplier itself is not a constant, but a function of your actual load.

The Relationship Between Throughput, Latency, and Concurrency

In inference systems, the token generation speed of a single request (a latency metric) and the total number of tokens per second produced by all requests on the whole server (a throughput metric) are two different quantities. As concurrency rises, the GPU must schedule computation across multiple sequences; the response speed of each request may drop slightly, but because otherwise idle compute time is filled, whole-machine throughput can rise multiplicatively. The numbers in the article illustrate exactly this: at 8 concurrency, a single request drops from about 648 to about 550 tokens/s, while whole-machine output jumps from about 590 to over 3,200 tokens/s. Therefore, deriving unit cost from "single-request speed ÷ rental" amounts to ignoring the fact that the GPU is shared across multiple requests in real high-concurrency scenarios, and naturally yields a figure far higher than reality.

The Three Variables That Determine Real Cost

The author summarizes the factors affecting cost into three points, each easy to overlook:

1. The server's real concurrency, not the concurrency achieved in a benchmark. Your business peak and average are what should drive pricing.
2. Idle time within the hour. The GPU is billed even while waiting, so cost per token should be "hourly rental ÷ all content the server actually produced in that hour," including idle periods.
3. Scale-to-zero cold-start overhead. On the same H100, a single cold wake-up took about 75 of 77.3 billed seconds before the server was ready to respond. This means elastic scaling is not a free lunch.

Scale-to-Zero and Cold Start

Scale-to-zero means scaling a GPU instance down to zero and stopping billing when there are no requests, then bringing it back up when new traffic arrives. It sounds like it can eliminate idle cost, but the restart process includes pulling container images, loading model weights from disk or network into VRAM, and initializing the CUDA context and inference engine, which can take anywhere from tens of seconds to several minutes. The example in the article is: of 77.3 billed seconds, about 75 were spent on cold start before the server was actually ready to respond—meaning you paid for that time but produced no tokens. Under per-second or per-hour GPU rental models, frequent cold starts drive cost back up. So whether to adopt scale-to-zero depends on whether your traffic is steady and sustained or low-frequency and bursty, and on the ratio of a single cold start's overhead to a single business volume.

The Ratio Holds on Smaller Cards Too

To verify the conclusion is not unique to the H100, the author reproduced it on a T4 with the same model and test method. At 8 concurrency, a single request decoded at 122–124 tokens/s, the whole server produced 701–708 tokens/s—again about 5.7x.

This shows that the cost-diluting effect of concurrency does not depend on any particular hardware generation.

Boundaries the Author Explicitly Draws

The author also emphasizes several limitations: this is a small model (0.5B), a single card, and only three runs per load type. With a 7B or 70B model, a different server, or longer prompts, the numbers will change.

I am not saying your cost is $0.22. I am saying that this number depends on your load, and only by measuring your own server can you know it.

The method, raw data, and open-source tooling have been published on GitHub (qaisermehdi3-coder/qvunex), and the author openly invites others' measured cost-per-token under different loads for cross-comparison.

Implications for Self-Hosted Inference Cost Accounting

The value of this data is not in the specific number $0.22, but in how it punctures a common assumption: that there is a stable conversion relationship between unit inference cost and hardware unit price. In reality, there is another layer in between: "utilization."

For teams doing private deployments or renting GPUs for inference, rather than agonizing over which card to buy or which cloud is cheaper, it is better to answer two questions first: What does your service's concurrency curve look like under real traffic? What proportion is idle time? These two answers may affect cost far more than switching vendors.

Feeding vLLM's /metrics endpoint into your own monitoring and regularly recording actual output tokens and billed duration is closer to the truth than any benchmark.

One-line Summary

Real-world measurements show that on the same H100, the cost per million tokens for single-request versus 8-concurrent inference differs by about 5.5x; cost depends on real load, not hardware unit price.

Paragraph Summary

A developer ran a controlled experiment with vLLM on an H100 on vast.ai: same card, same hourly rental ($2.689/hour), single-request serial inference came to about $1.26 per million output tokens, while 8-request concurrency dropped to about $0.22–0.23—a nearly 5.5x difference—with whole-machine throughput 5.65–5.84x the single-request perspective. The article points out that estimating cost as "single-request speed divided by hourly rental" overestimates by about 5.7x, because real cost is jointly determined by the server's actual concurrency, the idle share within the hour, and scale-to-zero cold-start overhead. The author reproduced the same ~5.7x ratio on a T4, and stresses that this is a 0.5B small-model single-card experiment; real cost must be measured against your own load.