The Optimal Decoder Is "Tilted Temperature": A New Theory of LLM Sampling Temperature

What should the sampling temperature of a large language model actually be set to? Experience tells us that the larger the sample budget, the higher the temperature should be. But the mathematical principle behind this intuition has long remained unclear. A new paper provides a surprising answer: the optimal decoding rule is a kind of "tilted temperature" sampling.

Starting from an overlooked contradiction

Temperature sampling is the most commonly used means of controlling randomness in LLM inference. With low temperature, outputs are conservative but prone to repetition; with high temperature, diversity increases but the model may start spouting nonsense. In real deployments, engineers often choose the temperature by trial and error, or rely on heuristic rules (such as raising the temperature as pass@k increases).

Behind this "tuning by feel" lies the absence of a unified optimization framework. The authors reformulate the decoding problem as pass@k maximization, subject to two constraints:

• Entropy lower bound: to prevent diversity collapse (the model repeatedly outputting the same answer)
• Cross-entropy upper bound: to prevent absurd outputs (sampling obviously incorrect paths)

These two constraints correspond exactly to the two most common failure modes in practical decoding. The question becomes: under this constrained optimization, what does the optimal sampling strategy look like?

pass@k is a common metric for measuring sampling quality in tasks such as code generation: for the same problem, sample k candidate answers independently, and success counts as long as any one of them passes verification; pass@k is the expected probability of that event. It naturally captures the ability to "get lucky by sampling several times," and is therefore closer to the real deployment workflow of "generate multiple candidates and then filter" than single-attempt accuracy. Precisely because it is sensitive to diversity among candidates, pass@k becomes dominated by high-probability paths as k grows, which explains why pursuing larger k often requires higher temperature—when diversity is insufficient, repeated sampling degenerates into hitting the same mediocre path over and over. Understanding this metric is the prerequisite for understanding the optimization objective in this paper.

The two constraints used in the paper come from information theory. Entropy measures how spread out a probability distribution is; an entropy lower bound requires that the sampling distribution not be overly concentrated, otherwise the model will repeatedly output the same answer, amounting to a collapse of diversity. Cross-entropy measures the difference between the sampling distribution and the true "correct" distribution; an upper bound limits how far sampling deviates from the correct answer, avoiding clearly incorrect paths. The two form a tension: raising entropy at all costs sacrifices correctness, while suppressing cross-entropy at all costs eliminates diversity. Writing these two quantities into the optimization problem as explicit constraints amounts to using a principled framework to characterize both failure modes simultaneously—"don't be too monotonous" and "don't be too absurd"—which is precisely the key distinction between this paper and empirical tuning.

Core conclusion: tilted temperature sampling

The paper proves that the above constrained optimization has a closed-form solution:

$$q^\star \propto (p \cdot e^{\eta V})^{1/T}$$

where $p$ is the model's original probability distribution, $V$ is a "verifier potential"—measuring the likelihood that the current partial generation will ultimately pass verification—$\eta$ is the constraint multiplier, and $T$ is the temperature.

The intuition behind this formula is that temperature sampling no longer uniformly "flattens" the entire distribution, but instead tilts it according to the likelihood that each path will ultimately succeed. Paths likely to succeed are relatively amplified, while paths doomed to fail are relatively suppressed.

The elegance of this framework lies in its unity: ordinary temperature sampling, verifier greedy decoding, and reward-tilted policies are all special cases of it. When $\eta = 0$, it degenerates to standard temperature sampling; when $T \to 0$ and only the highest $V$ value is retained, it becomes verifier greedy.

The theorem that temperature grows with k

The paper also has a result that directly addresses engineering intuition: under a rule where the diversity budget naturally expands with the number of samples $k$, the optimal temperature increases monotonically with $k$.

This means that "more samples, higher temperature" is no longer an empirical rule of thumb, but a provable theorem. The cost, however, is that the optimal temperature depends on the unknown verifier potential $V$, which cannot be computed directly in practice.

Two ways to bypass the verifier

The paper proposes two practical paths for estimating $V$:

Online setting—the J-EOS algorithm: It requires no correctness labels at all, jointly tracking the constraint multiplier and learning the verifier potential solely from binary pass/fail feedback. It is suitable for scenarios with interactive verification signals (such as code execution results or whether unit tests pass).

Offline setting—Calibrated Temperature Decoding (CTD): The paper finds that when the model is well calibrated, its log-probability and correctness are approximately affine-related. Under this condition, the tilt term collapses into the exponent, and the optimal strategy degenerates to ordinary temperature sampling. CTD only needs to fit a single scalar offline and incurs zero additional overhead at inference time.

The practical value of CTD is this: if the model is already calibrated, you can obtain near-optimal decoding performance without any extra computation—you just need to set that scalar temperature correctly.

Calibration in this paper refers to the degree to which a model's output probabilities match its actual accuracy. For a well-calibrated model, if the probability it gives for a certain type of problem is 0.8, then roughly 80% of such problems are indeed answered correctly. The paper finds that when a model is well calibrated, its log-probability and correctness are approximately affine-related (linear plus a constant), so the verifier potential that would otherwise need to be estimated separately can be absorbed into the temperature term, causing tilted sampling to degenerate into ordinary temperature sampling. CTD (Calibrated Temperature Decoding) exploits exactly this: it only requires fitting one scalar temperature offline, and no verifier queries or extra computation are needed at inference time. This also shows that model calibration itself is a hidden dividend that can be directly cashed in by the decoding strategy.

Validation on code generation

The authors evaluated the proposed methods on two code generation benchmarks, HumanEval and MBPP. The results show that the practical decoding algorithms based on these decision rules and CTD both outperform standard temperature sampling and adaptive decoding baselines. Code generation is a setting where verifier signals naturally exist (if the code runs, it passes), so the online learning of J-EOS has direct practical value.

Implications for practice

This work offers several actionable ideas for LLM inference optimization:

1. Temperature tuning is not an ever more mysterious art, under the pass@k objective it has a clear optimization form, and its growth with k is provable
2. Verifier signals are worth exploiting: if a task has automatic verification means (code execution, mathematical verification, format checking), J-EOS provides a learning framework that requires no labels
3. Model calibration is a hidden dividend: a well-calibrated model can directly use CTD with zero inference overhead
4. Do not neglect constraint modeling: the explicit modeling of the entropy lower bound and cross-entropy upper bound characterizes the decoding trade-off more precisely than a single temperature parameter

Of course, the quality of verifier potential estimation determines the upper bound of the method. Online J-EOS depends on sufficient pass/fail feedback, and offline CTD depends on the degree of model calibration—neither of which can always be satisfied in real deployment. The paper also leaves open questions: when the model is poorly calibrated and verification signals are sparse, how can a balance be struck between the two?

One-line Summary

A new paper proves that optimal LLM decoding is "tilted temperature" sampling, and that the optimal temperature increases monotonically with the number of samples k.

Paragraph Summary

The paper reformulates LLM decoding as a pass@k maximization problem under an entropy lower bound and a cross-entropy upper bound, and gives a closed-form optimal solution—"tilted temperature" sampling, which tilts the distribution according to the likelihood that each path will ultimately pass verification. This framework unifies ordinary temperature sampling, verifier greedy decoding, and reward-tilted policies. The paper proves that under the natural rule where the diversity budget expands with the number of samples k, the optimal temperature increases monotonically with k, turning the engineering experience that "more samples means higher temperature" into a theorem. To bypass the verifier potential that is difficult to compute directly, the authors propose two paths: online J-EOS (learning only from binary pass/fail feedback) and offline CTD (which degenerates to ordinary temperature sampling when calibration is good, with zero inference overhead), and validate on HumanEval and MBPP that they outperform standard and adaptive decoding baselines.