CUDA graph
A GPU execution optimization mechanism introduced in NVIDIA CUDA 10. A sequence of GPU operations is recorded in advance as a directed acyclic graph, which can then be replayed as a whole with a single submission, reducing CPU scheduling overhead. However, the recording phase requires the graph to run through first, and support for dynamic shapes is limited.
Timeline (last 90 days)
The core idea of the dlmrt inference engine is to capture the entire denoising loop into a single CUDA graph and execute it directly on the GPU, avoiding launching and synchronizing from the CPU at every step.
The 237.7-second prefill CUDA graph capture is the largest single block of overhead in the entire SGLang cold-start timeline.
At startup, vLLM typically performs CUDA graph capture only for the decode phase, and has a more aggressive trimming strategy for the set of captured batch sizes, so its cold-start graph capture phase is often shorter than SGLang's.
The cost of CUDA graphs is that the recording phase itself requires running the entire graph through once or even multiple times, and it has limited support for dynamic shapes and dynamic control flow.
In long-running services, the CUDA graph recording cost is amortized over tens of thousands of inferences and is nearly negligible; but in scale-to-zero cold-start scenarios, the recording happens only once yet must be put into use immediately, so it turns from a performance optimization into a warm-up tax.
Because the prefill phase must handle variable-length inputs, it often needs to capture CUDA graphs separately for multiple sequence lengths, further amplifying the recording time.
All Facts (6)
The core idea of the dlmrt inference engine is to capture the entire denoising loop into a single CUDA graph and execute it directly on the GPU, avoiding launching and synchronizing from the CPU at every step.
50%UnverifiedThe 237.7-second prefill CUDA graph capture is the largest single block of overhead in the entire SGLang cold-start timeline.
50%UnverifiedAt startup, vLLM typically performs CUDA graph capture only for the decode phase, and has a more aggressive trimming strategy for the set of captured batch sizes, so its cold-start graph capture phase is often shorter than SGLang's.
50%UnverifiedThe cost of CUDA graphs is that the recording phase itself requires running the entire graph through once or even multiple times, and it has limited support for dynamic shapes and dynamic control flow.
50%UnverifiedIn long-running services, the CUDA graph recording cost is amortized over tens of thousands of inferences and is nearly negligible; but in scale-to-zero cold-start scenarios, the recording happens only once yet must be put into use immediately, so it turns from a performance optimization into a warm-up tax.
50%UnverifiedBecause the prefill phase must handle variable-length inputs, it often needs to capture CUDA graphs separately for multiple sequence lengths, further amplifying the recording time.
50%