Prefill Stage
The prefill stage is the first stage of large-model inference, in which the model reads the entire input context at once and computes attention. Its computational load is approximately proportional to context length and directly affects first-token latency.
Timeline (last 90 days)
The prefill stage is the process in which the model builds a key-value cache (KV Cache) for all input tokens when processing user input.
Transformer inference is divided into two phases: a parallel compute-intensive prefill phase and a sequential memory-bandwidth-intensive decode phase.
All Facts (2)
The prefill stage is the process in which the model builds a key-value cache (KV Cache) for all input tokens when processing user input.
50%UnverifiedTransformer inference is divided into two phases: a parallel compute-intensive prefill phase and a sequential memory-bandwidth-intensive decode phase.
50%