Prefill-Decode Disaggregation
An architectural strategy in LLM inference systems that deploys the prefill stage and the decode stage on separate resources. By independently scaling compute for each stage, this approach optimizes overall inference throughput and latency.
Timeline (last 90 days)
PD disaggregation deploys the compute-intensive Prefill stage and the memory-bandwidth-intensive Decode stage of inference on separate instances for independent scaling and optimization
OpenAI has not adopted a prefill-decode separated PD architecture on its in-house chips
PD separation breaks data locality, as the KV cache generated by prefill nodes needs to be transferred to decode nodes over the network, with bandwidth consumption and latency increasing as input sequence length grows
Prefill-Decode (PD) disaggregation requires transferring KV Cache from the Prefill instance to the Decode instance; a CX7 NIC at full 400 Gbps IB delivers approximately 50 GB/s, and transferring 40 GB of KV Cache takes 0.8 seconds.
Splitwise, DistServe, and Mooncake are all representative implementations of prefill-decode (PD) disaggregation.
All Facts (5)
PD disaggregation deploys the compute-intensive Prefill stage and the memory-bandwidth-intensive Decode stage of inference on separate instances for independent scaling and optimization
50%UnverifiedOpenAI has not adopted a prefill-decode separated PD architecture on its in-house chips
50%UnverifiedPD separation breaks data locality, as the KV cache generated by prefill nodes needs to be transferred to decode nodes over the network, with bandwidth consumption and latency increasing as input sequence length grows
50%UnverifiedPrefill-Decode (PD) disaggregation requires transferring KV Cache from the Prefill instance to the Decode instance; a CX7 NIC at full 400 Gbps IB delivers approximately 50 GB/s, and transferring 40 GB of KV Cache takes 0.8 seconds.
50%UnverifiedSplitwise, DistServe, and Mooncake are all representative implementations of prefill-decode (PD) disaggregation.
50%