D-Flash: The Secret Behind 3.5x LLM Inference Speedup with Diffusion Drafting

Introduction: The Bottleneck of Speculative Decoding

If you've ever deployed a large language model, you're all too familiar with the latency of long chain-of-thought reasoning—the model generates one token after another during inference, and waiting tens of seconds is far from unusual. To address this, the industry introduced Speculative Decoding, which uses a lightweight Draft Model to first guess a sequence of tokens, then has the target model verify them in parallel, achieving lossless acceleration.

However, this approach has a real-world bottleneck: advanced methods like EAGLE3 still have an autoregressive drafting stage—the Draft Model continues to generate tokens one by one, slowly. The result is essentially "speedup that goes nowhere," since forward latency grows linearly with the number of draft tokens.

D-Flash offers a highly inspiring solution: transform the drafting process into fast diffusion parallel generation, spitting out an entire block of tokens in a single forward pass, fundamentally bypassing the token-by-token autoregressive overhead.

The Core Conflict: The Trade-off Between Quality and Parallelism

To understand D-Flash's value, we first need to clearly see the two competing forces it tries to reconcile.

Autoregressive LLMs guarantee generation quality, but they are memory-bound, have low GPU utilization, and are inherently serial. Diffusion Language Models (Diffusion LMs) naturally support parallel generation and have a clear theoretical speed advantage—unfortunately, open-source diffusion models generally fall short of autoregressive backbones in quality, and to maintain acceptable results they require stacking a large number of denoising steps, which paradoxically makes actual inference slower.

The core conflict is: how can we combine the high quality of autoregression with the parallel capabilities of diffusion?

D-Flash's approach is remarkably clever—let the diffusion model "take a step back" and, instead of vying for the lead generation role, focus solely on serving as a Draft Adapter. It leverages the rich hidden features inside the target model as conditioning, trading a very low drafting cost for a high acceptance rate.

System Architecture: A Target-Model-Driven Drafting Pipeline

The entire inference pipeline begins with the target model. After the prompt is fed in, the target model performs a standard prefill and uniformly extracts hidden states from several layers, from shallow to deep. These features are concatenated and then compressed into unified "target context features" through a lightweight projection layer.

The key design lies in this: these features are injected into the KV Cache of every Draft Layer, rather than merely being mixed into the embeddings as input. The draft layers internally use a bidirectional attention plus MLP structure, with each layer directly obtaining the target model's Key and Value projections, so that even at deep layers there is no signal decay.

During the drafting stage, existing tokens and the mask tokens to be predicted enter the Draft Layer in parallel, producing an entire block of tokens in a single forward pass. Finally, the target model's LM Head outputs the logits, which are handed off to speculative decoding for parallel verification—the entire process is lossless, and the final result is still determined by the target model.

Breaking Down the Four Core Innovations

Innovation One: Fast Diffusion Parallel Drafting

An autoregressive Drafter runs token by token, so forward latency increases linearly with the number of draft tokens. D-Flash switches to a fast diffusion paradigm, denoising all mask tokens within a single block simultaneously and generating the entire block in one forward pass.

The latency comparison is quite striking: EAGLE3 needs 12 milliseconds to generate 8 tokens, and 16 tokens shoots up to 25 milliseconds; D-Flash, using a 5-layer draft model, generates 16 tokens in just 6 milliseconds—even lower than EAGLE3's cost for 8 tokens. Its insensitivity to block size means you can confidently deepen the Draft Model to increase acceptance length without being held hostage by drafting latency.

Innovation Two: Target Feature KV Injection

This design directly determines whether acceptance length can keep growing as the Draft depth increases. EAGLE3's approach fuses target features with the draft embeddings once at the input layer, causing the information to gradually dilute with depth.

D-Flash treats the fused target context features as persistent context, injecting them directly into the Key and Value projections of every Draft Layer and storing them in the KV Cache for repeated use. This way, even if the Draft Model is deepened to 5 or 8 layers, the conditioning signal does not decay, the acceptance length scales effectively with the number of layers, and the speedup ratio naturally pulls ahead.

Innovation Three: Random Anchor Sampling

Standard block diffusion training uniformly splits the response into blocks and masks fixed positions, which does not match the actual situation at inference time. D-Flash instead randomly samples clean tokens from the response as anchors for each mask block, then masks the subsequent tokens to train parallel prediction.

This directly aligns with the inference scenario—the draft model always starts speculating from the token verified in the previous step. Randomizing the anchors also exposes the Draft Model to more diverse target context features, improving data utilization efficiency. Experiments show that this strategy significantly outperforms the fixed-blocking approach in acceptance length and speedup on Math500, HumanEval, and MTBench.

Innovation Four: Loss Decay Weighted Training

In speculative decoding, a prediction error at an early position within a block causes all subsequent tokens to be discarded, so the accuracy of early positions is especially important. D-Flash applies an exponentially decaying weight to the cross-entropy loss, decreasing the weight for a token at position K within the block by γ^K—the earlier the position, the greater the weight.

The training curves show that after introducing loss decay, the acceptance length on Math500 in the very first epoch is noticeably higher than the uniform-weight baseline, and this position-aware loss design not only accelerates convergence but also slightly improves final convergence quality.

SOTA Comparison and Cross-Model Generalization

Under the SGLang production serving framework, D-Flash was comprehensively compared against MTP on the Qwen3.5 series. On Math500, HumanEval, and MTBench, D-Flash leads across the board in both acceptance length and speedup ratio—Qwen3.5-9B achieves a 3.5x speedup on HumanEval, and the same-scale 4B and 35B-A3B models also show clear advantages.

More importantly, its generality shines: the approach extends to Qwen3.5-27B, Qwen3-Coder-Next, and even GPT-OSS 20B and 120B, while maintaining efficient acceleration. The consistent performance across models and tasks demonstrates that "fast diffusion drafting + target feature KV injection" is not a tuning trick tailored to a specific model, but a broadly applicable LLM inference acceleration solution.

Ablation Studies: Validating the Rationale of Every Design

The ablation studies reveal the actual contribution of each design, with several conclusions especially worth noting:

• More Draft layers isn't always better: The 8-layer model has a higher acceptance length, but drafting latency rises accordingly, causing the overall speedup ratio to fall back. The 5-layer configuration achieved the best average speedup across multiple benchmarks, indicating a non-monotonic return curve, with 5 layers being the precise balance point between quality and cost.

• Extracting more target features is better: Extracting 5 hidden features from the target model outperforms 3 across all benchmarks—on Math500, 5H achieves a 4.31 speedup while 3H is only 3.98. The cost is a linear increase in offline storage overhead, but this is entirely worthwhile relative to the inference gains.

• Block size matching is crucial: When both training and inference use 16, the Math500 speedup is significantly better than in mismatched cases. Interestingly, training with large blocks generalizes to inference with small blocks, but not the other way around—this means you can train a general-purpose Drafter with large blocks and dynamically reduce the block size at inference according to load to control verification cost.

• KV injection soundly beats input fusion: Whether for autoregressive or fast diffusion drafting, the KV injection approach outperforms input fusion. After removing all target context features, the speedup of a pure five-layer fast diffusion Drafter plummets to 2.65~3.73, clearly confirming the foundation of D-Flash's design—the target model understands itself best, and fully leveraging its internal signals is the key to achieving a high acceptance rate.

Limitations and Future Outlook

D-Flash currently still requires staged supervised fine-tuning and reinforcement fine-tuning, and has not yet achieved end-to-end joint optimization, which could further raise the acceleration ceiling. Draft quality also heavily depends on the target feature distribution, requiring re-adaptation when the training data or target model version changes. Adaptive block-size scheduling has not yet been implemented—if the block size could be dynamically adjusted based on online concurrency, there would be further room to improve throughput.

Nevertheless, D-Flash carves out a new positioning of the "lightweight Draft Adapter" for diffusion models in the field of inference acceleration. It proves that the diffusion paradigm need not compete head-on with autoregressive backbones on generation quality, but can instead complement them as an efficient parallel drafter. The potential of this direction is clearly far from being fully explored.