Temperature Parameter

When a large language model generates each token, it samples from a probability distribution; the temperature parameter controls the degree of randomness—higher temperature produces more diverse outputs, while lower temperature produces more deterministic ones.

The Softmax formula with temperature parameter T is P(xᵢ) = exp(zᵢ/T) / Σexp(zⱼ/T); when T→0 it degenerates to argmax, and when T→∞ it approaches a uniform distribution

The temperature parameter design in the language model sampling process means that the same prompt may produce completely different outputs across different calls, so retrying itself does not guarantee convergence.

temperature为0时退化为贪婪解码，每步选取概率最高的Token，但仅能保证单次模型版本内的确定性

Even with the LLM temperature set to 0, the same intent may still generate different SQL structures due to differences in prompt wording, context length, and prior tool call results.

An LLM outputs a probability distribution when generating each word, and the higher the temperature the more random the sampling; even if two assistants recall the same information sources, different temperature settings can lead to different wording or even a different order of companies mentioned.

Under a sampling strategy with a temperature parameter greater than 0, even if the input is exactly the same, a large model may produce different token sequences across multiple runs, thereby following different tool-calling paths and reasoning processes.

The temperature parameter T adjusts the sharpness of the logits distribution via softmax(logits/T): the lower T is, the closer it is to greedy decoding, and the higher T is, the closer it is to uniform sampling.

In the context of attention, the temperature parameter controls the sharpness of the softmax: low temperature makes attention concentrate on a few keys (low entropy, possibly overfitting), while high temperature makes the weights more uniform (high entropy, possibly under-focusing).

The temperature parameter T controls the smoothness of the softmax; a higher T value makes the probability distribution smoother and exposes more information about the teacher model's internal decision logic.

When T=1 the output is a standard softmax distribution; when T>1 the probability distribution becomes more uniform—the higher the temperature, the flatter the output distribution, and the more fine-grained information the student model captures.

In typical practice, the temperature parameter T is commonly set between 3 and 10; after distillation training is complete, the student model reverts to the standard T=1 setting at inference time.

When the probability distribution during the decoding phase fluctuates (such as when the temperature parameter is too high or the context matches the training distribution of a particular language), the model may slip into the semantic space of another language when predicting the next token.

Sampling parameters, especially temperature, significantly affect model output, and are set independently by each tool and almost never exposed to the user.

Common sampling strategies include top-k sampling (retaining only the k tokens with the highest probability), top-p sampling (dynamically selecting the minimum set of tokens whose cumulative probability just exceeds a threshold p), and the temperature parameter (controlling the sharpness of the distribution by scaling the logits).