Decoding Randomness

A mechanism whereby a large language model does not always select the highest-probability token when generating each token, but instead samples randomly according to the probability distribution. Common parameters include temperature, top-k and top-p, used to adjust the diversity and determinism of generated text. It is one cause of hallucination.