DFlash2 Actually Slower on Qwen3? The Truth Behind the 30% Throughput Drop in Speculative Decoding

Why Speculative Decoding Can Get "Slower the More You Optimize"

Speculative Decoding is supposed to be a powerful tool for accelerating large model inference. It has been one of the most important breakthroughs in LLM inference acceleration since 2023. Traditional autoregressive generation produces only one token at a time, requiring a full forward pass each time, resulting in low GPU utilization. The core idea behind speculative decoding is to use a smaller, faster draft model to "guess" multiple upcoming tokens, then have the target model verify all these candidate tokens in parallel in a single pass. Since the draft model has low inference cost, even if some predictions are rejected, the overall end-to-end latency is significantly reduced. This technique was first systematically proposed by Google DeepMind in the paper Fast Inference from Transformers via Speculative Decoding, and has since spawned multiple variant implementations including Medusa, EAGLE, and DFlash.

However, in real-world deployments, many developers have encountered a counterintuitive phenomenon — after enabling DFlash2 speculative decoding, the throughput of the Qwen3-27B model actually decreased, even falling below the performance without speculative decoding at all.

This article is based on real-world benchmarks conducted by a Bilibili content creator running DFlash2 on vLLM across 4× RTX 4090 GPUs. We'll dissect the true cause behind this "30% throughput loss" phenomenon and discuss the community's fix progress and optimization directions.

Benchmark Data: Fast on Single Connection, Collapses Under High Concurrency

The tests used a standard configuration of 1024 input tokens and 1024 output tokens, running the Qwen3-27B model on 4× RTX 4090 GPUs.

For single-connection performance, DFlash2's speed was acceptable: approximately 120 tokens per second, total tokens around 247, time-to-first-token around 300+ milliseconds, and per-token generation around 8 milliseconds. This is quite smooth for single-user interactive scenarios.

But as concurrency increased, things deteriorated. At around 52 concurrent connections, the system reached approximately 100% saturation, with a maximum of about 704 tokens per second and total tokens around 1440, but latency skyrocketed to a staggering 64 seconds. At 8 concurrent connections, the average throughput was about 588 tokens/second, with roughly 645 input tokens and 588 output tokens — already near the peak performance for this configuration.

Here's a telling detail — the draft acceptance rate was only about 41%: the draft model produced 1,055 tokens, but only about 438 were actually accepted. This acceptance rate directly determines whether speculative decoding can truly deliver acceleration.

The Real Culprit: Conflict Between Prefix Caching and Speculative Decoding

The core issue emerges when Prefix Caching and DFlash2 are enabled simultaneously. Prefix Caching is a classic optimization technique in LLM inference that reuses previously computed KV Cache. In multi-turn dialogue or batch processing scenarios, different requests often share the same system prompt or context prefix. Prefix caching stores the key-value pairs for these common segments so subsequent requests can load them directly without recomputation, saving 50%-80% of the prefill computation. Mainstream inference frameworks like vLLM enable this optimization by default, using a radix tree structure for fine-grained cache hit matching. However, there are fundamental design conflicts between cache granularity management and speculative decoding's verification logic — and this is precisely the root cause of the issue described in this article.

For models like Qwen3-27B, prefix caching is typically enabled by default to reuse previously computed KV Cache and avoid redundant computation.

However, vLLM's current implementation has a bug: in prefix caching, the system discards and recomputes the last cache unit (a mixed-layout block of approximately 16 tokens). This means that even when your request hits the cache, the tokens in the last unit still need to be recomputed.

Why Is the Last Cache Block Discarded?

The root cause lies in the verification mechanism of speculative decoding. Traditional speculative decoding needs to discard the unit containing the last hit token because the draft model's prediction at that position requires re-verification. Under DFlash2's mixed layout, the draft model generates blocks of 16 tokens at a time, so verification requires deleting and recomputing the entire 16-token cache block rather than just discarding a single token.

The mixed layout is a KV Cache organization method adopted by DFlash2 to improve memory efficiency, packing consecutive tokens into fixed-size cache blocks (typically 16 tokens). This design is more GPU-friendly for memory allocation and access patterns, but it also introduces management complexity: when speculative decoding verification fails and rollback is needed, the system cannot discard just a single token's cache — it must delete and recompute at the block level. When prefix caching is enabled, if the cache boundary happens to fall in the middle of a block, it triggers recomputation of the entire block, even though most tokens within it could have been reused. This granularity mismatch is what causes the performance regression described in this article.

This "context recomputation" problem means that when prefix caching + DFlash2 speculative decoding are both enabled, total throughput drops by approximately 37%, performing even worse than not using speculative decoding at all. Two optimizations that were each designed to accelerate inference end up dragging each other down when combined — this is the truth behind the "30% throughput loss."

Community Fix Progress: Bugfix Has Appeared, Awaiting Merge to Main Branch

vLLM is a high-performance LLM inference serving framework open-sourced by UC Berkeley, known for its innovative PagedAttention memory management and high throughput. It has become one of the mainstream choices for deploying large models in production environments. vLLM supports continuous batching, tensor parallelism, prefix caching, and many other optimization techniques, and was among the first to integrate speculative decoding implementations like DFlash2. Its modular design allows researchers to quickly experiment with new algorithms, but it also means that unforeseen interaction issues can arise when multiple optimizations are stacked. vLLM's GitHub repository is extremely active with rapid community response — the bugfix mentioned in this article was submitted by a community contributor within 4 days of the issue being reported.

The good news is that this issue has been acknowledged by the community on vLLM's GitHub project. The issue was formally reported approximately 4 days before testing, and the bugfix code was released the following day, targeting precisely the "context recomputation" problem described above.

However, as of testing time, this bugfix had not yet been merged into the main branch. Given the extremely fast iteration speed of the open-source community, this issue is expected to be resolved in vLLM's next release. After the fix, performance is expected to recover by 30%-40%, reaching the performance levels published in official benchmarks.

For teams currently using the Qwen3 + DFlash2 combination in production environments, it's recommended to closely monitor vLLM version updates, or carefully evaluate the combination of prefix caching and speculative decoding before the fix is merged.

Draft Acceptance Rate: The Key Variable That Determines Throughput Ceiling

Beyond the implementation-level bug, the DFlash2 algorithm itself is highly dependent on the draft acceptance rate. The acceptance rate is the core metric for speculative decoding performance, representing the proportion of candidate tokens generated by the draft model that pass the target model's verification. This metric directly determines the effective speedup ratio of speculative decoding: the higher the acceptance rate, the more effective tokens are "obtained" per verification round, and the lower the amortized computation cost per token. Theoretically, if the draft model's output distribution perfectly matches the target model, the acceptance rate can reach 100%, at which point the speedup ratio equals the speed advantage multiplier of the draft model. In practice, however, there's a capability gap between generic draft models and the target model, with acceptance rates typically falling between 40%-70%.

DFlash2 can generate a relatively large number of draft tokens at once (7 in the tests), but if the acceptance rate is too low, the waste during the verification phase drags down overall throughput.

Simple calculation: if the acceptance rate for 7 draft tokens can reach 50%-60%, that means accepting 3-4 tokens per round, which would significantly boost throughput. Official data claims DFlash2 averages 6 accepted tokens, but real-world measurements show this figure is clearly optimistic — at best around 58%-60%, and generally between 40%-50%, meaning 3-4 out of 7 tokens are actually accepted. This performance level is actually comparable to MTP (Multi-Token Prediction) approaches.

MTP (Multi-Token Prediction) is another implementation approach for speculative decoding, proposed by institutions including Meta. Unlike using an independent draft model, MTP trains the main model during the training phase to predict multiple future tokens simultaneously through multiple prediction heads that output in parallel. During inference, a single forward pass can directly generate multiple candidate tokens, which are then filtered through a verification mechanism. MTP's advantage is that it doesn't require a separate draft model, making deployment simpler; its disadvantage is that it requires modifying the model architecture and loss function during training, making it unfriendly to existing models. Performance-wise, MTP can achieve approximately 1.5-2× speedup at a 50% acceptance rate, comparable to well-tuned speculative decoding, but may underperform dedicated draft model approaches in extreme high-acceptance-rate scenarios.

Fine-tuning the Draft Model: Squeezing Out More Performance

To achieve higher throughput with DFlash2, improving the acceptance rate is crucial. There are already open-source projects attempting to retrain DFlash2's draft model, fine-tuning it with specific data to improve prediction hit rates.

For teams with their own business data, training or fine-tuning the draft model on domain-specific data is a clear optimization path: the better the draft model "understands" your data distribution, the higher the prediction hit rate, and the more pronounced the acceleration from speculative decoding. Key methods for improving acceptance rates include: using an early checkpoint of the target model as the draft model, fine-tuning the draft model on task-specific or domain-specific data, and adopting stronger small models (e.g., using Qwen2.5-7B as the draft model for Qwen3-27B).

Conclusion

DFlash2's stumble here is not a failure of the algorithm itself, but rather exposes the coupling trap between speculative decoding and existing optimizations like prefix caching at the engineering implementation level. It gives us two reminders:

First, stacking multiple inference optimizations doesn't necessarily yield additive gains. Cache management under mixed layouts requires extra caution, and real-world testing and validation cannot be skipped.

Second, the upper bound of speculative decoding's benefits is fundamentally determined by the draft acceptance rate. Generic draft models often fall short of the ideal values promoted in official benchmarks, and targeted fine-tuning is the key to unlocking real-world performance.

As the vLLM fix gets merged and the ecosystem for draft model fine-tuning matures, DFlash2 is poised to deliver on its promised acceleration value.