RangeSWA: Training-Free Length Extrapolation for Long-Text Generation in Diffusion Language Models

Continuous diffusion language models show promise in text generation tasks, but once the generation length exceeds the training length, problems such as text repetition, semantic drift, and degraded generation quality easily arise. To address these challenges, researchers have proposed a training-free length extrapolation method called RangeSWA, which combines local sliding window attention, long-range block summaries, and dynamic content retrieval to support long-range context access while preserving local details.

The Length Extrapolation Dilemma of Continuous Diffusion Language Models

Continuous Diffusion Language Models progressively denoise during generation and are capable of producing coherent text. However, these models typically only see sequences of a fixed length range during training, and when longer texts need to be generated at inference time, model performance degrades noticeably. Specifically: the generated text begins to repeat earlier segments, semantics gradually drift away from the initial topic, and overall quality declines as length increases.

The root of this problem lies in the fact that the model's positional encoding and attention mechanisms cannot properly generalize to lengths beyond the training distribution. Conventional solutions often require retraining or fine-tuning the model, which is costly and may damage existing capabilities. RangeSWA was proposed precisely to resolve this contradiction without updating any model parameters.

Positional Encoding and Length Extrapolation: Transformer-based models rely on positional encoding to distinguish tokens at different positions in a sequence. Common positional encodings (such as sinusoidal encoding, learnable absolute positional encoding, RoPE, etc.) only cover a limited length range during training, and if the sequence at inference exceeds that range, the positional representations enter numerical ranges the model has never seen, leading to misaligned attention allocation. The goal of length extrapolation research is to enable models to maintain reasonable behavior beyond the training length, with common approaches including position interpolation, position remapping, relative positional encoding modifications, and attention window constraints. The position remapping in RangeSWA is a concrete implementation of this idea: without changing model weights, it compresses or maps out-of-range position indices back into the training distribution, thereby avoiding positional encoding "overflow."

RangeSWA's Core Mechanism: Triple Attention and Triple Correction

RangeSWA's design can be broken down into three levels: "preserve locality, access long range, dynamic retrieval." Local sliding window attention ensures that the model can attend to nearby contextual details when generating each position; long-range block summaries compress distant historical information into a condensed representation to avoid attention dilution; and dynamic content retrieval extracts relevant segments from the historical context according to the needs of the current generated content.

Adjusting the attention mechanism alone is not enough. The research team also introduced three correction measures to mitigate the generation degradation caused by length extrapolation: positional remapping readjusts positional encodings beyond the training range so that they fall within a range the model can handle; conditioning attention correction calibrates the distribution of attention weights; and denoising-stage-adaptive window scheduling dynamically adjusts the window size according to different stages of the diffusion process.

According to the original paper, these components work together so that the model can maintain both local coherence and global semantic consistency during long-text generation.

Sliding Window Attention and Block Summaries: Standard self-attention lets each position attend to the entire history, with computation growing quadratically with sequence length, and long-distance attention is easily diluted by a large number of irrelevant tokens. Sliding Window Attention limits the visible range of each position to a nearby fixed window, significantly reducing computation and maintaining local coherence, but at the cost of losing long-distance information. Block Summary, on the other hand, compresses and encodes history outside the window block by block, forming a small number of summary tokens for subsequent positions to access, thereby preserving long-range information at lower cost. The combination of the two forms a hybrid context access pattern of "fine-grained local + coarse-grained long-range," which is also the key to RangeSWA's ability to balance detail and global semantics in long texts.

Experimental Validation: Both Perplexity and Repetition Rate Drop

Researchers conducted experiments on multiple continuous diffusion language models. The results show that RangeSWA can effectively reduce generative perplexity and text repetition rate in long-text generation. These two metrics directly reflect generation quality: the lower the perplexity, the more confident and reasonable the model's prediction of the next word; the lower the repetition rate, the better the diversity of the text.

Because the method is training-free, it can be directly applied to existing continuous diffusion language models without additional training data or computational resources. This is a significant advantage for practical deployment—developers can extend generation length by adjusting inference strategies without changing model weights.

Perplexity and Repetition Rate Metrics: Perplexity measures the model's predictive uncertainty over text; a lower value indicates a higher predicted probability for the true token and a more concentrated distribution, usually positively correlated with generation fluency. However, in long-text generation, low perplexity does not always equal high quality—the model may reduce perplexity by repeatedly outputting high-frequency segments, so it needs to be observed together with the repetition rate metric. Repetition rate typically measures the proportion of repeated n-grams or the coverage of repeated segments in the generated text, and is used to assess diversity. Only when both decrease simultaneously does it indicate that the model maintains predictive confidence without falling into a degeneration loop. This is also why the study chose these two metrics as the core basis for evaluation.

The Significance and Limitations of Training-Free Extrapolation

RangeSWA represents a practical path for inference optimization of diffusion language models. Compared with extrapolation methods that require retraining, the training-free approach lowers the barrier to use and avoids the risk of catastrophic forgetting. However, the original material does not disclose details of the method's performance under different model scales and different generation lengths, nor does it discuss comparisons with extrapolation methods for autoregressive language models.

Another issue worth noting is computational overhead. Sliding windows, block summaries, and dynamic retrieval all increase computation at inference time. Although training costs are eliminated, whether this overhead is acceptable in long-sequence scenarios still requires more practical testing to verify.

From a trend perspective, diffusion language models are moving from research to application, and length extrapolation is an unavoidable engineering problem. RangeSWA provides an idea for extending the generation window without changing the model. If its generality can be validated on more model architectures and its inference efficiency optimized in the future, such methods are expected to become a standard component in the deployment toolbox for diffusion language models.

One-line Summary

RangeSWA achieves length extrapolation for continuous diffusion language models in a training-free manner, reducing perplexity and repetition rate.

Paragraph Summary

The article introduces the problems of repetition, semantic drift, and quality degradation that arise in continuous diffusion language models when generation length exceeds the training length, and presents the training-free length extrapolation method RangeSWA. The method consists of triple attention mechanisms (local sliding window attention, long-range block summaries, dynamic content retrieval) and three correction measures (positional remapping, conditioning attention correction, denoising-stage-adaptive window scheduling), extending the generation window without updating model parameters. Experiments show that RangeSWA can simultaneously reduce generative perplexity and text repetition rate, and can be directly applied to existing models. The article also points out its limitations: it lacks detailed data under different model scales and generation lengths, does not compare with extrapolation methods for autoregressive models, and its inference computational overhead remains to be verified; generality and efficiency remain directions for future attention.