Frequency-Aware Remasking
A decoding strategy for diffusion language models that tracks how frequently each token prediction changes across denoising steps and prioritizes committing high-frequency-change tokens to encourage exploration of alternative reasoning paths and avoid collapse onto a single trajectory.
Timeline (last 90 days)
A paper proposes formalizing the test-time scaling of DLMs as the adaptive allocation of inference compute and decoding strategy, and introduces a frequency-aware remasking strategy.
This study designs a two-stage method: first use a fast base sampler to solve easy problems, then hand the remaining difficult problems to the frequency-aware remasking strategy.
Frequency-aware remasking tracks the frequency with which each token's prediction changes across denoising steps, and prioritizes committing those tokens that change frequently.
Frequency-aware remasking dynamically balances exploration (retaining uncertain tokens to try alternative paths) and exploitation (committing stable tokens to advance generation).
For problems that the base determinism-aware strategy still failed to solve after 100 samples, frequency-aware remasking was able to recover 3.75% to 18.75% of the cases.
Frequency-aware remasking provides a lightweight decoding strategy adjustment that improves reasoning capability without retraining the model.
Frequency-aware remasking is complementary to existing determinism-aware strategies and can be used in combination with them.
All Facts (7)
A paper proposes formalizing the test-time scaling of DLMs as the adaptive allocation of inference compute and decoding strategy, and introduces a frequency-aware remasking strategy.
50%UnverifiedThis study designs a two-stage method: first use a fast base sampler to solve easy problems, then hand the remaining difficult problems to the frequency-aware remasking strategy.
50%UnverifiedFrequency-aware remasking tracks the frequency with which each token's prediction changes across denoising steps, and prioritizes committing those tokens that change frequently.
50%UnverifiedFrequency-aware remasking dynamically balances exploration (retaining uncertain tokens to try alternative paths) and exploitation (committing stable tokens to advance generation).
50%UnverifiedFor problems that the base determinism-aware strategy still failed to solve after 100 samples, frequency-aware remasking was able to recover 3.75% to 18.75% of the cases.
50%UnverifiedFrequency-aware remasking provides a lightweight decoding strategy adjustment that improves reasoning capability without retraining the model.
50%UnverifiedFrequency-aware remasking is complementary to existing determinism-aware strategies and can be used in combination with them.
50%