BCSM: An Efficient New Objective Breaking Through the Bottleneck of Discrete Diffusion Distillation

Existing Challenges in Discrete Diffusion Distillation

Discrete diffusion models generate sequences through parallel decoding and show promise in modeling discrete data such as text and code. However, their inference requires multiple iterative denoising steps, resulting in high computational cost. Distillation techniques aim to compress multi-step diffusion models into few-step or even single-step generators, but existing methods have fundamental limitations.

The most common distillation objective is token-wise KL divergence, which exactly matches the product-form block distribution but may provide insufficient supervision signal in the probability-space optimization geometry when the number of distillation steps is very small. In addition, teacher-only candidate truncation further restricts the student model's learning space—the student can only imitate tokens that the teacher deems high-probability, ignoring other potentially reasonable options.

Concrete Score Matching (CSM) offers another approach: constructing an intra-position objective by aligning relative logit geometry. However, the full pairwise form of CSM has a computational complexity of O(|V|²), where |V| is the vocabulary size, making it intractable for large-vocabulary scenarios.

Discrete diffusion models are a class of generative models that generalize the noise-adding and denoising framework of diffusion models to discrete data (such as text tokens and code symbols). Unlike continuous diffusion, which adds Gaussian noise in real-valued space, discrete diffusion is typically defined on the probability simplex, performing forward corruption between discrete states via transition matrices (e.g., uniform noise or absorbing states), and learning to recover the original token sequence from the noise distribution during the reverse process. Inference requires multiple iterative denoising steps, with each step predicting all positions of the full sequence in parallel—this "parallel decoding" is its advantage over autoregressive models; however, when the number of steps is large, the total computation (measured by the number of neural function evaluations, NFE) remains significant, which is precisely the motivation for distillation compression.

Token-wise KL divergence is the most intuitive objective in discrete distillation: computing and summing the KL divergence between teacher and student distributions at each position. It is equivalent to matching the product-form block distribution, i.e., assuming conditional independence across positions. The problem is that when the student is compressed to very few steps (e.g., a single step), its parameter capacity and optimization path are constrained, and the gradients provided by token-wise KL tend to weaken or become unidirectional in both "nearly matched" and "severely mismatched" regions, leading to insufficient supervision signal. Concrete Score Matching (CSM) originates from the idea of discrete score matching: instead of directly matching probability values, it aligns the relative logit differences between adjacent candidates (i.e., "concrete scores"), thereby imposing constraints on the relative geometry of probability space with lower sensitivity to absolute probability scale. However, the full pairwise form requires traversing all word pairs, with complexity O(|V|²).

BCSM: Bidirectional Centered Score Matching

To address the above issues, the paper proposes Bidirectional Centered Score Matching (BCSM), an efficient intra-position distillation objective. Its core design consists of three aspects:

1. Constructing an O(K) Bidirectional Active Set

BCSM no longer considers all word pairs; instead, it constructs a bidirectional active set of size K for each position. This set simultaneously contains the teacher model's high-probability candidates and the student model's currently preferred candidates, forming a bidirectional supervision signal. This avoids the information loss caused by unilateral teacher truncation while compressing the number of candidates from |V| to K.

Active set is a key technique for sparsifying pairwise comparisons: not all tokens in the vocabulary are worth comparing—it suffices to align relative geometry among a few highly relevant candidates. The "bidirectional" aspect of BCSM means the set simultaneously incorporates the teacher model's high-probability candidates (ensuring teacher knowledge is not lost) and the student model's currently preferred candidates (ensuring the student can explore reasonable options beyond teacher truncation), with the union truncated to K. This preserves the teacher signal while leaving room for correcting the student's own distribution. K is typically much smaller than |V| (e.g., tens to hundreds), making the number of comparisons per position manageable, and after factorization the actual computation can be reduced to O(K).

2. Pairwise Comparison Reduced to O(K²) and Factorized

Within the active set, BCSM performs pairwise comparisons to align relative logit geometry, reducing complexity from O(|V|²) to O(K²). Furthermore, through mathematical factorization, the computational complexity of the final objective is reduced to O(K), making training overhead largely decoupled from vocabulary size.

3. Cross-Timestep Transition Alignment

In addition to the intra-position objective, BCSM introduces cross-timestep transition alignment to ensure that the student model's transition behavior between different denoising steps is consistent with the teacher. This is particularly critical for few-step distillation, as errors at each step accumulate.

Experimental Results: The Trade-off Between Quality and Efficiency

The paper validates the effectiveness of BCSM on multiple benchmarks. Under single-step inference settings, BCSM achieves significant efficiency improvements: up to 6× reduction in the number of neural function evaluations (NFE) compared to baselines, while achieving a 16.9% relative improvement on HumanEval and a 15.1% relative improvement on MMMU. These results demonstrate that BCSM can maintain generation quality under aggressive few-step distillation.

Compared to token-wise KL, BCSM's optimization geometry focuses more on relative ranking than absolute probability, which helps the student explore reasonable outputs beyond teacher truncation. Compared to full CSM, BCSM reduces complexity from quadratic to linear (with respect to K) through the active set and factorization, making the method practically scalable.

NFE (Number of Function Evaluations) is the standard metric for measuring the inference cost of diffusion/flow models, referring to the total number of times the denoising network (or its distilled student network) is invoked to generate a sample. Single-step distillation means NFE=1, i.e., a single forward pass produces the result. The "up to 6× NFE reduction" reported by BCSM refers to a substantial decrease in the number of network calls required compared to multi-step baselines while achieving comparable or better quality. HumanEval and MMMU are representative benchmarks for code generation and multimodal understanding, respectively; relative improvement refers to the percentage improvement over baseline methods under the same or fewer steps, demonstrating that aggressive compression does not sacrifice generation quality.

Implications for Discrete Diffusion Distillation

The value of BCSM lies not only in performance numbers but also in revealing a design principle: in few-step distillation, the quality of supervision signal matters more than quantity. Through the bidirectional active set and relative geometry alignment, BCSM provides more effective gradient signals with less computation. Cross-timestep alignment reminds us that distillation is not merely matching a single-step distribution but coordinating the entire denoising trajectory.

For researchers and engineers working on diffusion model acceleration, BCSM offers a new alternative to KL divergence, particularly suitable for scenarios with huge vocabularies and limited labeled data. Future work could explore adaptive selection of the active set size K and generalize BCSM to continuous diffusion models.

One-line Summary

BCSM reduces discrete diffusion distillation complexity to O(K) via bidirectional active sets and factorization, achieving up to 6× NFE reduction in single-step inference with quality improvements.

Paragraph Summary

The article addresses the high inference cost of discrete diffusion models, reviewing the limitations of existing distillation objectives: token-wise KL provides insufficient supervision signal under very few steps, teacher candidate truncation restricts student exploration, and Concrete Score Matching, while aligning relative logit geometry, has pairwise computation of O(|V|²). To this end, Bidirectional Centered Score Matching (BCSM) is proposed, achieving efficient intra-position distillation by constructing an O(K) bidirectional active set containing both teacher and student candidates, factorizing pairwise comparisons to O(K), and introducing cross-timestep transition alignment. Experiments show that under single-step inference, NFE is reduced by up to 6×, with a 16.9% relative improvement on HumanEval and 15.1% on MMMU. The article emphasizes that in few-step distillation, the quality of supervision signal matters more than quantity, and BCSM provides a new alternative to KL divergence for scenarios with huge vocabularies.