top-k sampling
A language model decoding strategy that retains only the k candidate tokens with the highest probabilities and then samples from among them, used to control the diversity of generation.
Timeline (last 90 days)
Nucleus sampling and top-k sampling are currently the two most commonly used heuristic strategies in LLM decoding, and their hyperparameters (k or p) are static and cannot adaptively adjust according to context complexity.
The fundamental difference between EOS and nucleus sampling and top-k sampling is that EOS does not control quality by truncating the distribution, but instead dynamically adjusts the temperature through entropy maximization under information-theoretic constraints, with guarantees from online learning theory.
Generation strategies such as temperature parameter, top-k sampling, and nucleus sampling are all means of adjusting and controlling the probability distribution.
Common sampling strategies include top-k sampling (retaining only the k tokens with the highest probability), top-p sampling (dynamically selecting the minimum set of tokens whose cumulative probability just exceeds a threshold p), and the temperature parameter (controlling the sharpness of the distribution by scaling the logits).
All Facts (4)
Nucleus sampling and top-k sampling are currently the two most commonly used heuristic strategies in LLM decoding, and their hyperparameters (k or p) are static and cannot adaptively adjust according to context complexity.
50%UnverifiedThe fundamental difference between EOS and nucleus sampling and top-k sampling is that EOS does not control quality by truncating the distribution, but instead dynamically adjusts the temperature through entropy maximization under information-theoretic constraints, with guarantees from online learning theory.
50%UnverifiedGeneration strategies such as temperature parameter, top-k sampling, and nucleus sampling are all means of adjusting and controlling the probability distribution.
50%UnverifiedCommon sampling strategies include top-k sampling (retaining only the k tokens with the highest probability), top-p sampling (dynamically selecting the minimum set of tokens whose cumulative probability just exceeds a threshold p), and the temperature parameter (controlling the sharpness of the distribution by scaling the logits).
50%