Spot Instance
A business model in which cloud providers sell idle computing resources at a discounted price while retaining the right to reclaim the resources at any time, usually giving a warning window of 30 seconds to 2 minutes, with price and availability highly dependent on regional supply and demand.
Timeline (last 90 days)
Spot instances are at peak during daytime on weekdays in the US and Europe, with tight capacity and frequent preemptions; late at night and on weekends, supply is ample and prices are lower.
Decision rules for judging whether Spot is suitable: batch inference, checkpoint-supporting or stateless workloads, tolerance of 5-10 minute interruptions, and experimental scenarios where cost comes first are suitable for Spot; latency SLAs for real users, model loading exceeding 2 minutes, US and Europe peak hours, and interruption costs exceeding the savings are suitable for on-demand instances.
For a Spot instance at $0.20/hour preempted 3 times per hour with a 70B model reloading for 4 minutes each time, the reload overhead is $0.04/hour and the effective rate is about $0.24/hour; but if preempted 10 times per hour, the effective rate rises to $0.33/hour.
Spot instances utilize idle capacity in cloud data centers, with prices typically 60%–90% lower than on-demand pricing, but they may be reclaimed at any time.
All Facts (4)
Spot instances utilize idle capacity in cloud data centers, with prices typically 60%–90% lower than on-demand pricing, but they may be reclaimed at any time.
60%UnverifiedSpot instances are at peak during daytime on weekdays in the US and Europe, with tight capacity and frequent preemptions; late at night and on weekends, supply is ample and prices are lower.
50%UnverifiedDecision rules for judging whether Spot is suitable: batch inference, checkpoint-supporting or stateless workloads, tolerance of 5-10 minute interruptions, and experimental scenarios where cost comes first are suitable for Spot; latency SLAs for real users, model loading exceeding 2 minutes, US and Europe peak hours, and interruption costs exceeding the savings are suitable for on-demand instances.
50%UnverifiedFor a Spot instance at $0.20/hour preempted 3 times per hour with a 70B model reloading for 4 minutes each time, the reload overhead is $0.04/hour and the effective rate is about $0.24/hour; but if preempted 10 times per hour, the effective rate rises to $0.33/hour.
50%