Stop Overpaying for GPUs: Spot vs On-Demand Instance Cost Comparison for LLM Inference

If you're running large model inference on rented GPUs, chances are you're overpaying — not by a little, but by 2-3x. The problem isn't which cloud provider you choose; it's which payment model you pick.

A practitioner who has long tracked GPU cloud pricing laid out a practical comparison framework for Spot vs On-Demand instances in their analysis, using concrete numbers to reveal where most people waste the most money.

The Price Gap Is Much Larger Than You Think

Take the RTX 4090 (24GB), the workhorse for 7B-32B model inference:

• On-Demand instances: Roughly $0.34–$0.74/hour, depending on the cloud provider
• Spot/preemptible instances: The same card at roughly $0.15–$0.30/hour

Same silicon, 50%-70% price difference. The A100 80GB is similar: about $1.20/hour on-demand vs. roughly $0.50/hour on Spot.

But cheap never comes without a cost.

How Spot Instances Actually Perform in LLM Inference

Spot instances can be reclaimed by the cloud provider with very short notice (typically 30 seconds to 2 minutes). For LLM inference, this creates three practical problems:

1. Model reload time: Loading a 70B model into VRAM takes 3-5 minutes. If you get preempted twice an hour, the losses can exceed the savings.
2. Request interruptions: In-flight requests fail outright, requiring retry logic or a queue mechanism to handle fallback.
3. Availability fluctuations: Spot capacity dries up during peak hours, and the "cheap" GPU you need may simply not exist.

How Spot instances (preemptible instances) work: Spot instances are a business model where cloud providers sell idle compute at a discount. The provider reserves the right to reclaim at any time, typically giving a 30-second to 2-minute warning window before reclamation. The essence of this mechanism: cloud providers prioritize capacity for On-Demand instances, and Spot instances only exist when there's surplus compute. As a result, Spot pricing and availability are highly dependent on regional supply and demand — weekday daytime in the US and Europe is peak hours with tight capacity and frequent preemptions; late nights and weekends offer ample supply and lower prices. Understanding this mechanism helps you judge: when your inference workload runs and how much interruption you can tolerate directly determine whether Spot is really right for you.

Decision Framework: When to Use Which

Scenarios Suited for Spot

• Batch inference (embedding, batch classification, offline evaluation)
• Workloads that support checkpointing or are inherently stateless
• Can tolerate 5-10 minute interruptions
• Running experiments where cost matters more than latency

Scenarios Suited for On-Demand

• Facing real users with latency SLA requirements
• Model load time exceeds 2 minutes (large models)
• During US/Europe peak hours when Spot market supply is thin
• When interruption cost exceeds the savings — this math needs to be done carefully

The Hybrid Strategy Most People Should Use

Run 1-2 On-Demand instances for baseline traffic, scale with Spot for burst traffic. Most inference workloads have a predictable baseline and a peak ceiling. Pay full price for the baseline, Spot prices for the peaks.

The Math Everyone Skips: Effective Hourly Cost

Don't just compare hourly rates — calculate the effective cost per thousand tokens:

effective_hourly = (hourly_rate × hours_run + reload_cost) / hours_run

Where reload_cost = preemptions × reload_time_hours × hourly_rate.

Example: $0.20/hour Spot, preempted 3 times per hour, 70B model reloading takes 4 minutes each time —

• Reload overhead: 3 × (4/60) × $0.20 = $0.04/hour
• Effective rate: $0.24/hour, still far below the $0.74 On-Demand price

But what if you're preempted 10 times per hour? The effective rate rises to $0.33/hour, plus the experience loss from latency. At that point, On-Demand actually becomes the better deal.

The value of this formula is converting the intuition of "cheap" into comparable numbers. Many teams look only at the sticker price when choosing Spot, ignoring the hidden costs of reloading and interruptions.

Effective cost per thousand tokens: The effective hourly cost formula given in the article only considers the time dimension, but the ultimate billing unit for LLM inference is tokens. Dividing the effective hourly cost by the number of tokens produced in that hour (i.e., throughput) gives you the true cost per thousand tokens. Throughput varies enormously across different models and batch sizes: a 7B model processing 32 concurrent requests on a 4090 might produce hundreds of thousands of tokens per hour; a 70B model doing single-stream inference might have throughput an order of magnitude lower. This means the same card in different configurations can have per-thousand-token costs that differ by several times. After comparing Spot vs On-Demand, it's worth using per-thousand-token cost to horizontally compare different model sizes and deployment options, avoiding optimization on the wrong dimension.

The Easily Overlooked Hidden Cost: Egress Fees

The most easily overlooked cost isn't compute — it's data transfer out. Some cloud providers charge $0.05-0.09/GB for egress. If you're running RAG with large context windows or serving multimodal models, egress fees can exceed the GPU bill itself. Before making decisions, be sure to check egress pricing.

Why Egress Fees Exist and How to Avoid Them: Cloud providers charge for data flowing out of their network boundary, while inbound traffic is typically free. The business logic behind this design is customer lock-in — data comes in easily but is hard to get out, raising migration costs. In LLM inference scenarios, egress fees mainly arise in three areas: returning inference results to users, pulling RAG retrieval content from external data sources (if cross-cloud), and downloading model weights or logs. Mitigation strategies include: deploying inference services in the same region or even the same availability zone as the caller to use internal networking, caching high-frequency results at the edge, and factoring egress unit pricing into total cost of ownership (TCO) comparisons when selecting cloud providers. Some smaller cloud providers use low egress fees as a differentiator, which is worth including in your vendor selection considerations.

Practical Recommendations

Choosing between Spot and On-Demand is fundamentally a trade-off between cost and reliability. There are three core principles:

• Calculate effective cost, not sticker price: Plug reload time and preemption frequency into the formula to get the true hourly cost.
• Hybrid deployment beats choosing one: Use On-Demand for baseline stability, Spot for peak cost reduction.
• Don't forget egress fees: They can be a major line item in RAG and multimodal scenarios.

Cloud GPU pricing changes fast — it's worth verifying each provider's latest prices directly before making a final decision.

One-line Summary

Spot instances can save 50-70% on GPU costs, but you need to account for the hidden costs of reloading and interruptions.

Paragraph Summary

The article points out that when renting GPUs for large model inference, most people overpay by 2-3x due to poor payment model choices. Using RTX 4090 and A100 as examples, Spot instances are 50%-70% cheaper than On-Demand, but the trade-off is a 30-second to 2-minute reclamation warning, causing model reloads (70B models need 3-5 minutes) and request interruptions. The article provides a decision framework: batch, stateless, interruption-tolerant workloads suit Spot; latency-SLA workloads, large model loading, and peak hours suit On-Demand; most scenarios should adopt a hybrid strategy with On-Demand for baseline and Spot for peaks. The core method is calculating effective hourly cost by plugging preemption frequency and reload time into the formula, rather than just comparing sticker prices. Additionally, egress fees ($0.05-0.09/GB) can exceed the GPU bill in RAG and multimodal scenarios and should be verified in advance.