Greedy Decoding
An inference strategy for language models in which the token with the highest probability is selected at each generation step. It corresponds to sampling with temperature set to zero and theoretically produces deterministic outputs.
Core Facts
Timeline (last 90 days)
Greedy decoding always selects the token with the highest probability, while random sampling draws from the probability distribution, which means the same prompt may yield different responses.
The non-determinism of large language models stems from their autoregressive sampling process; even with temperature set to 0, differences in inference frameworks, batch sizes, and hardware floating-point operations can all cause output drift.
All Facts (2)
The non-determinism of large language models stems from their autoregressive sampling process; even with temperature set to 0, differences in inference frameworks, batch sizes, and hardware floating-point operations can all cause output drift.
75%UnverifiedGreedy decoding always selects the token with the highest probability, while random sampling draws from the probability distribution, which means the same prompt may yield different responses.
60%