Decode

The Prefill phase is a compute-intensive operation, while the Decode phase is a memory-bandwidth-intensive operation.

The Prefill phase is a compute-intensive task with high demands on compute power; the Decode phase is a memory-access-intensive task that is sensitive to GPU memory bandwidth.

The prefill stage of Transformer inference is primarily bottlenecked by compute (FLOPS), while the decode stage is primarily bottlenecked by memory bandwidth.

TTFT is determined by the Prefill phase and is compute-bound, while TPOT corresponds to the Decode phase and is memory-bandwidth-bound.

The Prefill stage of LLM inference is compute-intensive with GPU utilization close to 100%, while the Decode stage is memory-bandwidth-intensive with the bottleneck being HBM VRAM bandwidth.

传统混合部署中Prefill阶段的大批量计算会阻塞Decode阶段的token生成，导致TTFT波动剧烈

LLM inference during the decode phase is typically memory-bandwidth-bound rather than compute-bound, because token-by-token generation requires loading the entire model weights but has extremely low arithmetic intensity.

LLM 推理分为预填充阶段（Prefill，计算密集型）和解码阶段（Decode，内存带宽密集型）两个阶段

Large model inference is divided into two phases, Prefill and Decode; Prefill is a major component of Time to First Token (TTFT), while Decode corresponds to Time Per Output Token (TPOT).

During the Decode phase of large models, because each step depends on the result of the previous step and can only be executed serially, hardware utilization is typically below 10%.

Disaggregated inference decouples the compute-intensive prefill stage from the memory-bandwidth-intensive decode stage, allocating them to separate hardware resource pools best suited to each for independent scaling.

The prefill stage is highly parallel with large throughput but latency that grows with context length, while the decode stage generates tokens auto-regressively and its speed is limited by memory bandwidth.

The Prefill (prompt processing) phase is compute-intensive and relies on the GPU's parallel matrix computation capability; the Decode (generation) phase is memory-bandwidth-intensive and depends more on VRAM bandwidth.

The prefill phase requires processing all input tokens at once and is a compute-intensive operation, whereas the decode phase generates only one token per step and is a memory-bandwidth-intensive operation.

The Pre-Fill phase of AI inference is computationally intensive and more reliant on matrix computation capability; the Decode phase is memory-bandwidth-bound for single users with small batch sizes

The Neural Accelerator's speedup for Prefill is approximately 3–5 times, far exceeding its speedup for Decode, which benefits more from increased memory bandwidth.

The Decode stage produces only one token at a time, resembling GEMV in form, with extremely low arithmetic intensity and is memory-bound.

AI Creative Insights was launched by the Entropik team, who are also the developers of the Decode platform.

The Decode phase of TPS is the response speed actually perceived by the user, and hardware memory bandwidth is often the true bottleneck of the decoding phase rather than pure floating-point compute.

Separating Prefill and Decode to run on different resources is the mainstream architecture for optimizing resource utilization in current large-scale inference services.