PACT: A Certified Commit Rule for Parallel Decoding in Diffusion Language Models

Diffusion Language Models accelerate generation by writing multiple tokens in a single forward pass through parallel decoding. However, existing commit rules decide which tokens to write by testing only the predicted distribution, ignoring the reliability of the tokens actually written after sampling. This article introduces PACT, a training-free commit rule that samples first and then certifies, ensuring higher quality of written tokens and thereby achieving equal or better answer quality with fewer forward passes.

The Bottleneck of Parallel Decoding: Commit Rules Disconnected from Sampling

Masked diffusion language models (such as the LLaDA and Dream series) can predict tokens at multiple positions in parallel when generating text. To decide which positions can be written simultaneously, researchers have proposed various commit rules, such as those based on the confidence or entropy of the predicted distribution. These rules typically assume that if the model's predicted distribution at a position is sufficiently certain, that position can be safely committed.

But the problem is that in sampling-based decoding, the tokens actually written are randomly drawn from the predicted distribution, and the commit rule does not consider these actually drawn tokens when testing. In other words, the rule only cares about the shape of the distribution, not the content ultimately written down. This leads to a key contradiction: the quality of the answer actually depends on the reliability of the written tokens, not on the reliability of the distribution itself.

The authors found that the log-probability of an answer can be decomposed into two parts: the reliability of the written tokens, and the dependencies within each step. Existing commit rules focus only on the former (through distribution testing) while ignoring the latter (the mutual influence among actually written tokens). This explains why parallel decoding sometimes produces repetitive or contradictory text.

Masked Diffusion Language Model is a class of text generation models based on the diffusion process. Its core idea is: during training, some tokens in the sequence are randomly masked, and the model learns to recover the original tokens from the masked sequence; during inference, it starts from a fully masked sequence and progressively "unmasks" through multiple rounds of iteration, predicting and writing tokens at multiple positions in parallel each time. Unlike autoregressive models that generate token by token from left to right, masked diffusion models allow multiple positions to be updated simultaneously in a single forward pass, thus offering the potential for parallel decoding. The LLaDA and Dream series are representative works in this direction; they typically adopt the Transformer architecture and control the trade-off between generation quality and speed by adjusting the masking ratio and sampling strategy. Understanding the iterative unmasking mechanism of masked diffusion is the prerequisite for understanding why commit rules are critical: which positions are written in each round directly determines the context of subsequent rounds and the final sequence quality.

PACT: Sample First, Then Certify

To address the above problem, the paper proposes PACT (Parallel Decoding with Certified Tokens), a training-free commit rule. Its core idea is intuitive: first sample the tokens to be written, then certify whether this set is reliable, and finally space out the positions written at each step in the sequence.

Specifically, the PACT process consists of three steps:

1. Sample: According to the current predicted distribution, sample a token for each candidate position.
2. Certify: Check whether the sampled token set satisfies some reliability condition (for example, each token's probability at its position exceeds a certain threshold). Only certified tokens are written.
3. Space: Spread out the positions written at each step in the sequence rather than committing them contiguously. This reduces dependency interference between positions and improves generation quality.

The key advantage of this certification mechanism is that it directly guarantees the reliability of the written tokens rather than relying solely on the shape of the distribution. The paper proves that PACT's certification holds at every temperature, and that no rule can commit more positions under the same guarantee. This means PACT is theoretically optimal (under the certification constraint).

Commit rule is the decision mechanism in parallel decoding used to determine, after each round of forward propagation, which positions' predictions are sufficiently reliable to be written into the sequence immediately and which positions need to be left for subsequent rounds of refinement. Common commit rules are based on the confidence of the predicted distribution (e.g., maximum probability exceeding a threshold) or entropy (e.g., distribution entropy below a threshold), with the intuition that the more concentrated the distribution, the more certain the model, and the lower the writing risk. However, in stochastic sampling decoding, the token ultimately written is a sample drawn from the distribution, not the distribution itself; a high-confidence distribution can still sample a low-probability token. PACT's certification step targets exactly this gap: it first fixes the sampling result, then checks whether the probability of that specific token at its position meets the threshold, thereby upgrading "distribution reliability" to "written content reliability." This order of sampling before certification is the most essential difference between PACT and existing rules.

Experiments: Fewer Forward Passes, Higher Quality

The authors conducted experiments on four models, including the LLaDA and Dream families, using five benchmarks: HumanEval, MBPP, GSM8K, MATH500, and IFEval. These benchmarks cover tasks such as code generation, mathematical reasoning, and instruction following.

Experimental results show that PACT requires fewer forward passes to achieve the same quality as published commit rules. Although the spacing strategy reduces the number of positions committed per step, subsequent steps can therefore write more tokens, and the total cost is usually lower. In addition, at the same speed, published decoders have a much higher probability of writing the same word twice in a row than PACT, indicating that PACT effectively reduces the problem of repetitive generation.

These results support the paper's core claim: commit rules should certify the tokens they actually write, not merely test the predicted distribution.

The number of forward passes mentioned in the article is a core metric for measuring the inference cost of diffusion language models. Unlike autoregressive models, which require one forward pass per token, diffusion models can process the entire sequence in parallel in each round of forward propagation, but require multiple rounds of iteration to complete generation. Therefore, the total cost is approximately "number of rounds × per-round forward overhead." The role of the commit rule is to compress the number of rounds: the more positions committed per round, the fewer rounds required, but committing too early may introduce erroneous tokens and contaminate subsequent context. PACT's spacing strategy appears to reduce the commit volume per round, but by reducing dependency interference between positions, it enables subsequent rounds to commit in batches more safely, thereby gaining an advantage in total rounds. In the experiments, HumanEval and MBPP evaluate code generation, GSM8K and MATH500 evaluate mathematical reasoning, and IFEval evaluates instruction following; these tasks are sensitive to token reliability and can therefore effectively distinguish the strengths and weaknesses of commit rules.

Implications and Outlook

PACT offers a new approach for parallel decoding in diffusion language models: tightly integrating the commit rule with the sampling process to ensure the reliability of written content. This method requires no additional training and can be directly applied to existing models, offering good practical value.

In the future, further exploration can be made into optimizing the certification conditions and extending PACT to a broader range of generation tasks. The paper's code has been open-sourced, and interested readers can visit the project page for more details.

One-line Summary

PACT proposes a sample-then-certify commit rule that improves the quality and efficiency of parallel decoding in diffusion language models.

Paragraph Summary

This article addresses the disconnect between commit rules and sampling in parallel decoding of diffusion language models and proposes the training-free PACT method. Existing commit rules test only the shape of the predicted distribution while ignoring the reliability of the tokens actually sampled and written, leading to unstable parallel decoding quality. PACT changes the process to first sample tokens, then certify the reliability of the set, and finally space out the write positions, directly guaranteeing the reliability of written content, and theoretically proves that it commits the optimal number of positions under the certification constraint. Experiments on four models including LLaDA and Dream and five benchmarks show that PACT achieves equal or better quality with fewer forward passes and significantly reduces the problem of consecutive repeated words. The code has been open-sourced.