Greedy Decoding
An inference strategy for language models in which the token with the highest probability is selected at each generation step. It corresponds to sampling with temperature set to zero and theoretically produces deterministic outputs.
Core Facts
The non-determinism of large language models stems from their autoregressive sampling process; even with temperature set to 0, differences in inference frameworks, batch sizes, and hardware floating-point operations can all cause output drift.
The temperature parameter scales the Logits vector, and when temperature is 0, it degenerates into greedy decoding.
Timeline (last 90 days)
Greedy decoding is extremely prone to falling into degenerate loops on low-bit quantized models, because weight quantization loses a large amount of numerical precision and the model's scores for certain token sequences exhibit pathologically high peaks, causing the same passage of text to be output repeatedly.
Greedy decoding always selects the token with the highest probability, while random sampling draws from the probability distribution, which means the same prompt may yield different responses.
All Facts (4)
The non-determinism of large language models stems from their autoregressive sampling process; even with temperature set to 0, differences in inference frameworks, batch sizes, and hardware floating-point operations can all cause output drift.
75%VerifiedThe temperature parameter scales the Logits vector, and when temperature is 0, it degenerates into greedy decoding.
65%UnverifiedGreedy decoding always selects the token with the highest probability, while random sampling draws from the probability distribution, which means the same prompt may yield different responses.
60%UnverifiedGreedy decoding is extremely prone to falling into degenerate loops on low-bit quantized models, because weight quantization loses a large amount of numerical precision and the model's scores for certain token sequences exhibit pathologically high peaks, causing the same passage of text to be output repeatedly.
50%