[KongchangAI]
· 1 min read· 926 words

Long-Text Generation Collapses at High Temperatures? The Sampler Is the Key

Long-Text Generation Collapses at High Temperatures? The Sampler Is the Key

Research finds sampler candidate pool size at high temperatures determines long-text generation success, with top-nsigma far outperforming top-p.

A study of 29 open-source models (1B–122B) reveals that long-text generation degradation may stem from sampler choice rather than model capability. At higher temperatures, samplers keeping a small stable candidate pool (like top-nsigma) retain accuracy, while those admitting low-probability tokens (top-p, top-k, Top-H) collapse—top-p solved 0 of 100 GitHub issues at temperature 2 versus 46–62 for top-nsigma. Candidate pool size correlates with quality at Spearman -0.77.

Open-weight language models are known to degrade when generating long texts — this is almost consensus. But the common remedy of simply scaling up training may overlook a critical inference-time variable: the sampler. A recent study systematically tested 29 open-source models (1B to 122B parameters) across ten benchmark tasks and a million-token-level needle-retrieval task, revealing a strong link between sampling strategy and long-text generation quality.

Research on sampling issues in long-context generation

How the Sampler Determines the Success of Long Generation

The study covered scenarios including SWE-bench Verified coding agents, long chain-of-thought reasoning, long-form writing, long structured outputs, and long-context reading. The core finding: as temperature rises, samplers that keep the candidate token pool small and stable do not lose accuracy, while samplers that allow low-probability tokens into the candidate pool degrade sharply.

At temperature 2, coding agents using top-p, top-k, or Top-H solved 0 out of 100 GitHub issues; under the same conditions, top-nsigma solved 46 to 62, and won all issues where the two samplers disagreed (based on three Qwen3.5 models). Across all benchmarks, top-nsigma retained a median of 100% of its temperature-1 performance, while top-p retained only 26%.

The Relationship Between Temperature and the Sampler

Temperature is a hyperparameter that scales the logits distribution during language model decoding: temperature T divides the raw logits by T before applying softmax. T<1 makes the distribution sharper (more deterministic, more conservative), T=1 preserves the model's original distribution, and T>1 flattens the distribution, giving originally low-probability tokens more chances to be selected. Thus temperature itself does not directly determine the sampling outcome — it changes the shape of the probability distribution, while the actual "filtering rule" is handled by the sampler.

Common samplers include: top-k (keep only the k tokens with the highest probabilities), top-p / nucleus sampling (keep the smallest set of tokens whose cumulative probability reaches p), min-p (keep tokens whose probability is no lower than some fraction of the highest probability), Top-H (dynamically determine the candidate set size based on entropy), and top-nsigma discussed in this article (determine the candidate pool by a multiple of standard deviations from the mean). Different samplers behave very differently as temperature rises: some rapidly expand the candidate pool to tens of thousands of tokens, while others maintain a small, stable set. This is precisely the mechanistic basis for the finding that "the sampler determines the success of long generation" — temperature is merely an amplifier, and the actual size of the candidate pool is what directly affects generation quality.

Candidate Pool Size Explains the Collapse

The study provides an intuitive explanation by counting the number of tokens each sampler admits. In one Llama-3.1-8B case, as temperature rose from 1 to 2, top-p's candidate pool ballooned from 4 tokens to 51,845, while top-nsigma stayed at 3.

Across 133 sampler settings and temperature combinations, typical candidate pool size predicted judge quality with a Spearman correlation of -0.77. More importantly, if the candidate pool is fixed at temperature 1 and the temperature is adjusted afterward, Top-H, min-p, and p-less no longer collapse.

What the Spearman Correlation Coefficient Means

The Spearman rank correlation coefficient is a non-parametric statistic that measures the strength of a monotonic relationship between two variables, ranging from -1 to +1. It does not require data to be normally distributed or assume a linear relationship; instead, it first converts each set of observations into ranks and then computes the Pearson correlation between the ranks. The Spearman correlation of -0.77 reported here between candidate pool size and judge quality means there is a strong negative monotonic relationship: the larger the candidate pool, the lower the generation quality ranking. A value close to -1 indicates that this single simple statistic — candidate pool size — can predict the quality ordering across sampler configurations quite accurately, without relying on complex internal model metrics. For engineering practice, this means "whether the candidate pool expands as temperature rises" can serve as a quick diagnostic signal for choosing a sampler.

Performance at Default Temperature and Implications

At the default temperature of 1, most samplers perform similarly, meaning the sampler's impact is masked in everyday use. But once temperature is raised to increase diversity or exploration, the choice of sampler becomes critical.

This study sends a clear signal: long-context generation degradation is not necessarily due to insufficient model capability — sampling strategy may be a more direct and lower-cost intervention point. The team has released all code and data. For agent applications that rely on high-temperature sampling, rethinking sampler design may be more effective than additional training.

One-line Summary

Research finds that at high temperatures, the sampler's candidate pool size determines long-text generation success, with top-nsigma far outperforming top-p.

Paragraph Summary

A systematic study of 29 open-source models (1B to 122B parameters) reveals that degradation in long-text generation among open-weight language models may not stem from insufficient model capability but may be closely tied to inference-time sampler choice. Covering SWE-bench coding agents, long-chain reasoning, long-form writing, structured outputs, and million-token needle retrieval, the study finds that as temperature rises, samplers that keep the candidate token pool small and stable (such as top-nsigma) do not lose accuracy, while samplers that allow low-probability tokens into the candidate pool (such as top-p, top-k, Top-H) degrade sharply. At temperature 2, top-p and others solved 0 out of 100 GitHub issues, while top-nsigma solved 46 to 62. The Spearman correlation between candidate pool size and generation quality is -0.77. At the default temperature of 1, most samplers perform similarly; differences emerge only in high-temperature exploration scenarios.

#sampler#long-text generation#top-nsigma#top-p#temperature#open-weight language models
Share:

Related articles