Disaggregated Serving
An advanced deployment architecture for large model inference that separates the Prefill and Decode stages onto distinct GPU nodes. This separation prevents mutual interference between the two workload types and improves overall GPU utilization. It requires dedicated high-speed transfer mechanisms to move KV Cache data between nodes.
Timeline (last 90 days)
Disaggregated inference decouples the compute-intensive prefill stage from the memory-bandwidth-intensive decode stage, allocating them to separate hardware resource pools best suited to each for independent scaling.
In disaggregated inference, the KV cache generated by prefill nodes must be transferred to decode nodes in real time, and for long-context or large-batch requests a single transfer may reach several GB.
Disaggregated deployment physically separates the prefill and decode phases, handing each to a purpose-built runtime, so that each phase runs at its own optimal operating point.
Traditional inference runs the Prefill stage and Decode stage serially on the same group of GPUs, causing the two types of workloads to interfere with each other and making it difficult to optimize GPU utilization
All Facts (4)
Traditional inference runs the Prefill stage and Decode stage serially on the same group of GPUs, causing the two types of workloads to interfere with each other and making it difficult to optimize GPU utilization
70%UnverifiedDisaggregated inference decouples the compute-intensive prefill stage from the memory-bandwidth-intensive decode stage, allocating them to separate hardware resource pools best suited to each for independent scaling.
50%UnverifiedIn disaggregated inference, the KV cache generated by prefill nodes must be transferred to decode nodes in real time, and for long-context or large-batch requests a single transfer may reach several GB.
50%UnverifiedDisaggregated deployment physically separates the prefill and decode phases, handing each to a purpose-built runtime, so that each phase runs at its own optimal operating point.
50%