D-Flash: How Diffusion Drafting Delivers 3.5x Faster LLM Inference

Introduction: The Bottleneck of Speculative Decoding

If you've ever deployed a large language model, you're intimately familiar with the latency of long chain-of-thought reasoning—during inference, the model generates one token after another, and waits of tens of seconds are far from unusual. To address this, the industry introduced Speculative Decoding, which uses a lightweight draft model to guess a sequence of tokens first, then has the target model verify them in parallel, achieving lossless acceleration.

However, this approach has a real-world bottleneck: advanced methods like EAGLE3 still have an autoregressive drafting phase—the draft model still generates tokens one at a time. The result is "acceleration that goes nowhere," with forward latency growing linearly with the number of draft tokens.

D-Flash offers a highly instructive solution: transform the drafting process into fast diffusion parallel generation, spitting out an entire block of tokens in a single forward pass, fundamentally bypassing the token-by-token autoregressive overhead.

The Core Tension: Balancing Quality and Parallelism

To understand the value of D-Flash, we first need to clearly see the two competing forces it attempts to reconcile.

Autoregressive large models offer guaranteed generation quality, but they're memory-bound, suffer from low GPU utilization, and are inherently serial. Diffusion language models (Diffusion LMs) natively support parallel generation, with a clear theoretical speed advantage—but unfortunately, open-source diffusion models generally fall short of autoregressive backbones in quality. To maintain acceptable results, they need to stack numerous denoising steps, which actually makes real-world inference slower.

The core tension is: how do you combine the high quality of autoregression with the parallel capability of diffusion?

D-Flash's approach to solving this is remarkably clever—it lets the diffusion model "take a step back" and not compete for the primary generation role, but instead focus on serving as a Draft Adapter. It leverages the rich hidden features inside the target model as conditioning, trading an extremely low drafting cost for a high acceptance rate.

System Architecture: A Target-Model-Driven Drafting Pipeline

The entire inference pipeline begins with the target model. After receiving the prompt, the target model performs a standard prefill and uniformly extracts hidden states from several layers, from shallow to deep. These features are concatenated and passed through a lightweight projection layer, compressing them into a unified "target context feature."

The key design lies here: this feature is injected into the KV Cache of every Draft Layer, rather than merely being mixed into the embedding as input. The draft layers internally use a bidirectional attention plus MLP structure, and every layer directly accesses the target model's Key and Value projections—so the signal doesn't attenuate even in the deeper layers.

During the drafting phase, existing tokens and the mask tokens to be predicted enter the Draft Layer in parallel, generating the entire block's tokens in a single forward pass. Finally, the target model's LM Head outputs logits, which are handed off to speculative decoding for parallel verification—the entire process is lossless, and the final result is still determined by the target model.

Breaking Down the Four Core Innovations

Innovation One: Fast Diffusion Parallel Drafting

An autoregressive drafter runs token by token, with forward latency increasing linearly with the number of draft tokens. D-Flash switches to a fast diffusion paradigm, denoising all mask tokens within a single block simultaneously, generating the whole block in a single forward pass.

The latency comparison is striking: EAGLE3 needs 12 milliseconds to generate 8 tokens, jumping to 25 milliseconds for 16 tokens; D-Flash generates 16 tokens using a 5-layer draft model in just 6 milliseconds—even lower than the cost of EAGLE3 generating 8 tokens. Its insensitivity to block size means you can confidently deepen the draft model to increase acceptance length without being constrained by drafting latency.

Innovation Two: Target Feature KV Injection

This design directly determines whether acceptance length can continue to grow as the draft depth increases. EAGLE3's approach fuses the target features with the draft embedding once at the input layer, so the information gradually dilutes as the layers deepen.

D-Flash treats the fused target context feature as persistent context, directly injecting it into the Key and Value projections of every Draft Layer and storing it in the KV Cache for repeated use. This way, even if the draft model is deepened to 5 or 8 layers, the conditioning signal doesn't attenuate, the acceptance length effectively scales with the number of layers, and the speedup ratio naturally widens the gap.

Innovation Three: Random Anchor Sampling

Standard block diffusion training splits the response into uniform blocks and masks at fixed positions, which doesn't match the actual situation at inference time. D-Flash instead randomly samples clean tokens from the response as anchors for each mask block, then masks the subsequent tokens to train parallel prediction.

This directly aligns with the inference scenario—the draft model always starts its speculation from the tokens verified in the previous step. Randomizing anchors also exposes the draft model to more diverse target context features, improving data utilization efficiency. Experiments show that this strategy significantly outperforms fixed-block schemes in both acceptance length and speedup on Math500, HumanEval, and MTBench.

Innovation Four: Loss Decay Weighted Training

In speculative decoding, a prediction error at an early position within a block invalidates all subsequent tokens, so accuracy at early positions is especially important. D-Flash applies exponentially decaying weights to the cross-entropy loss, decreasing tokens at position K within the block by γ^K—the earlier the position, the greater the weight.

Training curves show that after introducing loss decay, the acceptance length on Math500 in the first epoch is noticeably higher than the uniform-weight baseline. Moreover, this position-aware loss design not only accelerates convergence but also achieves a slight improvement in final convergence quality.

SOTA Comparison and Cross-Model Generalization

Under the real-world SGLang serving framework, D-Flash was comprehensively compared with MTP on the Qwen 3.5 series. On Math500, HumanEval, and MTBench, D-Flash leads across the board in both acceptance length and speedup—Qwen 3.5-9B achieves a 3.5x speedup on HumanEval, and the same-scale 4B and 35B-A3B also show clear advantages.

More importantly, its versatility: the approach extends to Qwen 3.5-27B, Qwen 3-Coder-Next, and even GPT-OSS 20B and 120B, still maintaining efficient acceleration. The consistent performance across models and tasks demonstrates that "fast diffusion drafting + target feature KV injection" isn't a tuning trick tailored to a specific model, but a universal LLM inference acceleration solution.

Ablation Studies: Validating the Rationale of Each Design

The ablation studies reveal the actual contribution of each design, with several conclusions particularly worth noting:

• More draft layers aren't always better: The 8-layer model has a higher acceptance length, but drafting latency also rises accordingly, causing the overall speedup ratio to decline. Five layers achieve the best average speedup across multiple benchmarks, indicating a non-monotonic returns curve—5 layers is the precise balance point between quality and cost.

• Extracting more target features is better: Extracting 5 hidden features from the target model performs better than 3 across all benchmarks—on Math500, 5H achieves a speedup of 4.31, while 3H is only 3.98. The tradeoff is a linear increase in offline storage overhead, but it's completely worth it given the inference gains.

• Block size matching is critical: When both training and inference use 16, the Math500 speedup is significantly better than in mismatched cases. Interestingly, training with large blocks can generalize to inference with small blocks, but not vice versa—meaning you can train a general-purpose drafter with large blocks and dynamically reduce the block size at inference time based on load to control verification cost.

• KV injection thoroughly beats input fusion: Whether autoregressive or fast diffusion drafting, the KV injection approach outperforms input fusion. After removing all target context features, the speedup of a pure 5-layer fast diffusion drafter drops sharply to 2.65~3.73, clearly confirming the foundation of D-Flash's design—the target model understands itself best, and fully leveraging its internal signals is key to achieving a high acceptance rate.

Limitations and Future Outlook

D-Flash currently still requires staged supervised fine-tuning and reinforcement fine-tuning, and hasn't yet achieved end-to-end joint optimization, which could potentially further raise the acceleration ceiling. Draft quality also depends heavily on the target feature distribution, requiring re-adaptation when the training data or target model version changes. Adaptive block size scheduling hasn't been implemented yet—if the block size could be dynamically adjusted based on online concurrency, there would be further room to improve throughput.

Nevertheless, D-Flash opens up a new role for diffusion models in the inference acceleration domain: the "lightweight Draft Adapter." It proves that the diffusion paradigm doesn't need to compete head-on with the autoregressive backbone on generation quality, but can instead complement it as an efficient parallel drafter. The potential of this direction is clearly far from being fully explored.