Temperature Sampling
Timeline (last 90 days)
Multilingual language identification models must accommodate both high-resource and low-resource languages; common practices include stratified sampling by language, oversampling low-resource languages, or using temperature sampling to balance batches.
Ordinary temperature sampling, verifier greedy decoding, and reward-tilted policies are all special cases of the tilted temperature sampling framework. When η=0 it degenerates to standard temperature sampling; when T→0 and only the highest V values are retained it becomes verifier greedy.
Under the constrained optimization framework, the inverse temperature parameter of temperature sampling corresponds exactly to the Lagrange multiplier of the cross-entropy budget constraint.
Previous decoding methods (such as top-k, top-p, and temperature sampling) are mostly based on simplified assumptions about the logit distribution, but these assumptions are rarely verified.
Research directions for improving model calibration include introducing uncertainty tokens through supervised fine-tuning, performing post-calibration on probability distributions using temperature sampling, and training models to proactively output the ability to refuse to answer.
All Facts (5)
Multilingual language identification models must accommodate both high-resource and low-resource languages; common practices include stratified sampling by language, oversampling low-resource languages, or using temperature sampling to balance batches.
50%UnverifiedOrdinary temperature sampling, verifier greedy decoding, and reward-tilted policies are all special cases of the tilted temperature sampling framework. When η=0 it degenerates to standard temperature sampling; when T→0 and only the highest V values are retained it becomes verifier greedy.
50%UnverifiedUnder the constrained optimization framework, the inverse temperature parameter of temperature sampling corresponds exactly to the Lagrange multiplier of the cross-entropy budget constraint.
50%UnverifiedPrevious decoding methods (such as top-k, top-p, and temperature sampling) are mostly based on simplified assumptions about the logit distribution, but these assumptions are rarely verified.
50%UnverifiedResearch directions for improving model calibration include introducing uncertainty tokens through supervised fine-tuning, performing post-calibration on probability distributions using temperature sampling, and training models to proactively output the ability to refuse to answer.
50%