scale-to-zero
A resource scheduling strategy used in cloud-native and on-demand GPU platforms in which backend worker instances are destroyed to release GPU resources when there are no requests, and instances are relaunched when new requests arrive. Idle time incurs no cost but increases cold-start latency.
Timeline (last 90 days)
On the same H100, a scale-to-zero cold wake-up took about 75 of 77.3 billed seconds before the server was capable of responding.
In long-running services, the CUDA graph recording cost is amortized over tens of thousands of inferences and is nearly negligible; but in scale-to-zero cold-start scenarios, the recording happens only once yet must be put into use immediately, so it turns from a performance optimization into a warm-up tax.
Scale-to-zero is a resource scheduling strategy for cloud-native and GPU on-demand platforms: when there are no requests, worker instances are destroyed and do not occupy GPUs, and when new requests arrive, instances are spun up again.
The author recommends that when running scale-to-zero services, three time points be recorded: billing start to health check passing, billing start to the first completed request, and the idle tail after the last request, and that pricing be based on the entire billed duration.
All Facts (4)
On the same H100, a scale-to-zero cold wake-up took about 75 of 77.3 billed seconds before the server was capable of responding.
50%UnverifiedIn long-running services, the CUDA graph recording cost is amortized over tens of thousands of inferences and is nearly negligible; but in scale-to-zero cold-start scenarios, the recording happens only once yet must be put into use immediately, so it turns from a performance optimization into a warm-up tax.
50%UnverifiedScale-to-zero is a resource scheduling strategy for cloud-native and GPU on-demand platforms: when there are no requests, worker instances are destroyed and do not occupy GPUs, and when new requests arrive, instances are spun up again.
50%UnverifiedThe author recommends that when running scale-to-zero services, three time points be recorded: billing start to health check passing, billing start to the first completed request, and the idle tail after the last request, and that pricing be based on the entire billed duration.
50%