RTX 3090 vs L40S Benchmark: How Batch Size Flips Performance and Cost

Why There's No Single "Which Is Faster" Answer

"Which is faster, the 3090 or the L40S?" — if you're expecting a simple multiplier, this benchmark will disappoint you. On machines rented via vast.ai, the author used the same vLLM 0.27.1 script, the same models (Qwen2.5 1.5B and 7B, including AWQ quantized), 128 output tokens, and enabled CUDA graphs. Throughput was tested at batch 1, 8, 32, and 128, with 3 repetitions per setting and the median taken. The result is clear: the performance comparison depends entirely on the number of concurrent requests.

At batch 1, the 3090 is roughly 1.03–1.09x the L40S; at batch 128 with the 7B AWQ model, the 3090 is only 0.46x the L40S. The reasoning: at small batch sizes the bottleneck is memory bandwidth (936 GB/s for the 3090 vs 864 GB/s for the L40S), so the two are roughly even; at large batch sizes compute density rises and the L40S's compute advantage is fully unleashed.

vLLM is an open-source high-throughput LLM inference and serving engine. Its core innovation is PagedAttention — managing the KV cache in pages like an operating system manages virtual memory, reducing memory fragmentation and thereby supporting larger batch sizes. vLLM supports continuous batching, dynamically inserting new requests into an in-flight batch to significantly boost GPU utilization. CUDA graphs, mentioned here, is a technique that records a sequence of kernel launches as a graph and submits it at once, reducing CPU launch overhead — especially beneficial for small-batch, low-latency scenarios. AWQ (Activation-aware Weight Quantization) is a 4-bit weight quantization method that preserves accuracy by protecting weight channels with a large impact on activations, greatly reducing memory footprint — but it still requires dequantization during compute, which can shift where the bottleneck lies across different hardware.

Performance Data: From Even to Far Behind

The author provides detailed ratios across three models (3090 speed ÷ L40S speed):

• 7B fp16: batch 1 = 1.03x, batch 8 = 1.04x, batch 32 = 1.11x, batch 128 drops sharply to 0.72x
• 7B AWQ: batch 1 = 1.09x, batch 8 = 1.01x, batch 32 = 0.71x, batch 128 = just 0.46x
• 1.5B fp16: batch 1 = 1.09x, batch 128 = 0.70x

You may not have noticed: with 7B AWQ, the 3090 is already clearly behind at batch 32 (0.71x), while the fp16 model isn't overtaken until batch 128. This shows that quantized models are more sensitive to changes in compute demand — AWQ reduces memory footprint but also makes compute a more prominent bottleneck. The author notes that variation across repeated runs was no more than 0.5% on the 3090 and at worst 2.6% on the L40S, so the data isn't noise.

The Cost Reversal: 3090 Cheaper at Low Batches, L40S Better at High Batches

Based on the prices the author actually paid ($0.414/hr for the 3090, $0.826/hr for the L40S), counting only GPU busy time, the cost per million output tokens is:

Model/Batch	3090 Cost	L40S Cost
7B fp16 batch 1	$2.22	$4.59
7B fp16 batch 128	$0.041	$0.059
7B AWQ batch 1	$0.77	$1.67
7B AWQ batch 128	$0.040	$0.036

At low batch sizes, the 3090 costs about half as much as the L40S; but at batch 128 with 7B AWQ, the L40S is already cheaper than the 3090 ($0.036 vs $0.040). Which card is cheaper depends on your traffic pattern, not the card itself. If your service is dominated by low-concurrency requests, the 3090 offers outstanding value; if you're after high throughput, the L40S's cost per token overtakes it.

Memory bandwidth refers to how fast the GPU reads data from memory, in GB/s; compute usually refers to FP16/FP32 floating-point throughput, in TFLOPS. During the decode phase of autoregressive inference, generating each token requires reading all model weights and the KV cache, so when the batch is small, compute units sit largely idle and performance is limited by memory bandwidth; as the batch grows, the weights read each time can be reused across multiple requests, compute density rises, and compute becomes the bottleneck. The 3090's memory bandwidth (936 GB/s) is slightly higher than the L40S's (864 GB/s), but the L40S's FP16 compute is far higher than the 3090's — which explains why the two are close at small batches and the L40S pulls ahead at large batches. Quantized models (such as AWQ) reduce weight size, further easing bandwidth pressure and making the compute bottleneck appear earlier.

Two Easily Overlooked Variables

The author highlights two points:

First, testing one card doesn't represent all cards of the same model. Another 3090 he had tested previously was clearly slower on every setting with the same script and driver. When choosing hardware, it's best to test two cards, not one.

Second, all cost figures assume the GPU is always busy. A real server can't be at full load all the time. If the GPU is only processing requests 30% of the time, the actual cost per token is about 3.3x the numbers above. This idle share may be the biggest number on your bill, and many monitoring dashboards don't show it at all. The author provides a 2-minute guide to checking your own idle share by reading /metrics.

GPU utilization usually refers to the proportion of time within a sampling period during which the GPU has at least one kernel executing; it can be observed indirectly via nvidia-smi or an inference framework's /metrics endpoint (such as the gpu_cache_usage_perc and num_requests_running exposed by vLLM). Note, however, that high utilization doesn't mean the compute units are saturated — waiting on data transfers from memory also shows up as "busy." So when evaluating cost, distinguish between "the GPU is occupied" and "compute is being effectively used." For cloud GPUs billed by the hour, idle time is billed too; actual cost per token = nominal cost ÷ utilization. The author suggests monitoring request queue length and batch size to estimate the real busy ratio, so hardware choices can be compared more accurately.

Implications for Real-World Hardware Selection

The value of this benchmark isn't in concluding that "the 3090 is faster than the L40S" or vice versa — it's in revealing an overlooked dimension: batch size completely changes the conclusion of a hardware comparison. For individual developers or small-scale services, the 3090 delivers comparable performance at half the cost in low-concurrency scenarios; for online services needing high throughput, the L40S is not only faster at batch 128 but also cheaper with AWQ models.

The author has released the data from all 192 runs, the scripts, and machine details, and offers paid custom benchmarking. If you're choosing hardware, start by asking yourself: what's my typical concurrency? How high is my GPU's actual utilization? The answers to these two questions will determine your bill far more than any single-card benchmark.

One-line Summary

The inference performance comparison between the 3090 and L40S reverses with batch size, and which is cheaper depends on your concurrency pattern.

Paragraph Summary

Using the same vLLM script on vast.ai, this benchmark of the RTX 3090 and L40S finds that the performance comparison depends entirely on concurrent request count: at batch 1 the 3090 is slightly faster (1.03–1.09x), while at batch 128 with the 7B AWQ model it's only 0.46x the L40S. On cost, the 3090's cost per token is about half the L40S's at low batches, but at high batches the L40S overtakes it on AWQ models. The author notes that small batches are bandwidth-bound while large batches unleash the L40S's compute advantage, and that quantized models make the compute bottleneck appear earlier. In addition, cards of the same model can vary in performance, and GPU idle time can significantly inflate real costs. Hardware selection should be based on typical concurrency and actual utilization, not single-card benchmarks.