Decode

The Prefill phase is a compute-intensive operation, while the Decode phase is a memory-bandwidth-intensive operation.

P/D Disaggregation (Prefill/Decode Disaggregation) deploys the Prefill and Decode stages on different nodes to independently scale according to their respective computational characteristics, while simultaneously optimizing throughput and time to first token (TTFT).

The Prefill phase is a compute-intensive task with high demands on compute power; the Decode phase is a memory-access-intensive task that is sensitive to GPU memory bandwidth.

During autoregressive decoding, each newly generated token requires reading the full KV Cache, making the decoding stage almost always memory-bandwidth-bound rather than compute-bound.

Large model inference is divided into two phases, Prefill and Decode; Prefill is a major component of Time to First Token (TTFT), while Decode corresponds to Time Per Output Token (TPOT).

The prefill stage of Transformer inference is primarily bottlenecked by compute (FLOPS), while the decode stage is primarily bottlenecked by memory bandwidth.

TTFT is determined by the Prefill phase and is compute-bound, while TPOT corresponds to the Decode phase and is memory-bandwidth-bound.

The Prefill stage of LLM inference is compute-intensive with GPU utilization close to 100%, while the Decode stage is memory-bandwidth-intensive with the bottleneck being HBM VRAM bandwidth.

传统混合部署中Prefill阶段的大批量计算会阻塞Decode阶段的token生成，导致TTFT波动剧烈

LLM inference during the decode phase is typically memory-bandwidth-bound rather than compute-bound, because token-by-token generation requires loading the entire model weights but has extremely low arithmetic intensity.

LLM 推理分为预填充阶段（Prefill，计算密集型）和解码阶段（Decode，内存带宽密集型）两个阶段

During the Decode phase of large models, because each step depends on the result of the previous step and can only be executed serially, hardware utilization is typically below 10%.

The inference framework uses a read-write separation architecture: the Prefill stage uses ordinary 8-card machines and the Decode stage uses large machines with 64 or 128 cards; the minimum configuration is 64 read cards and 64 write cards, 128 cards in total, and reads and writes can be deployed in different data centers connected by fiber optics.

The Decode phase is memory-bandwidth-intensive, requiring model weights and the KV cache to be reloaded from HBM at each step, which determines inter-token latency (ITL).

The prefill phase performs one forward computation over the entire input to produce KV caches for all positions, and is compute-bound; the decode phase generates tokens autoregressively one by one and is memory-bandwidth-bound.

Disaggregated inference decouples the compute-intensive prefill stage from the memory-bandwidth-intensive decode stage, allocating them to separate hardware resource pools best suited to each for independent scaling.

The prefill stage is highly parallel with large throughput but latency that grows with context length, while the decode stage generates tokens auto-regressively and its speed is limited by memory bandwidth.

The Prefill (prompt processing) phase is compute-intensive and relies on the GPU's parallel matrix computation capability; the Decode (generation) phase is memory-bandwidth-intensive and depends more on VRAM bandwidth.

The prefill phase requires processing all input tokens at once and is a compute-intensive operation, whereas the decode phase generates only one token per step and is a memory-bandwidth-intensive operation.

The Pre-Fill phase of AI inference is computationally intensive and more reliant on matrix computation capability; the Decode phase is memory-bandwidth-bound for single users with small batch sizes