DeepSeek Speculative Decoding Algorithm Merged into vLLM Main Branch — Single-User Token Speed Jumps Dramatically

Latest Developments in DeepSeek's Speculative Decoding Algorithm

DeepSeek's previously released speculative decoding algorithm (referred to in this article as "DSpark" / "DeepSpark") has reached a major engineering milestone — it has been officially merged into the vLLM project's main branch. This means inference images built on vLLM (such as the lightlayer builder image series) now natively support this speculative decoding approach without any additional configuration.

For those deploying open-source large language models, this is a noteworthy signal. Speculative Decoding is one of the most practically valuable inference acceleration techniques to emerge in recent years, and its adoption in mainstream inference frameworks directly determines whether everyday users can enjoy its performance benefits at low cost. This merger into vLLM's main branch marks the algorithm's official transition from the "research paper / reference implementation" stage to "production-ready."

Technical Background: What Is Speculative Decoding? Speculative decoding was formally introduced by the Google Brain team in 2023. Its core insight stems from a key observation: the inference bottleneck in large language models is not raw computation, but memory bandwidth — generating each token requires loading billions of parameters from GPU memory into compute units, a serial process that is deeply inefficient. Speculative decoding breaks this bottleneck by introducing a "Draft Model." The draft model is typically 1/10th the size of the main model or smaller, and can cheaply generate multiple candidate token sequences at high speed. The main model then performs a single parallel forward pass over the entire sequence, probabilistically accepting or rejecting each draft token. Since the cost of parallel verification is similar to verifying a single token, yet each verification pass can "confirm" multiple tokens, the theoretical speedup is 2–4×. DeepSeek's DSpark algorithm builds on this foundation with engineering optimizations tailored to its MLA (Multi-head Latent Attention) architecture, designing a dedicated draft strategy that achieves higher token acceptance rates in real deployments.

Framework Background: Why Does Merging into vLLM's Main Branch Matter? vLLM (Very Large Language Model Serving) is a high-performance inference framework open-sourced by UC Berkeley's LMSYS team in 2023. Its core innovation, PagedAttention, draws on the OS virtual memory paging concept to manage KV Cache, dramatically improving GPU memory utilization from under 40%. vLLM has since become one of the de facto standard frameworks for open-source LLM deployment. Code merged into the main branch means the feature has undergone review by core maintainers, passed CI/CD automated testing pipelines, and is committed to long-term maintenance — rather than sitting in an isolated branch or an experimental PR. This is the critical engineering signal that separates research output from production-ready capability.

Currently Supported Models

DSpark's speculative decoding algorithm currently supports the Qwen3 series and Google's Gemma series models.

Qwen3 is Alibaba Cloud's next-generation open-source LLM series released in 2025, featuring a hybrid thinking mode (supporting switching between streaming chain-of-thought and fast response) and MoE architecture variants. It has performed strongly across multiple benchmarks and quickly became one of the most downloaded model families in the open-source community. The Gemma series is Google DeepMind's lightweight open-source model line, known for excellent inference efficiency and a permissive commercial license, making it particularly well-suited for edge deployment. DSpark's prioritization of these two series reflects the vLLM community's strategic thinking on adaptation priorities — both have the largest user bases and most active community contributors, enabling the fastest test coverage and feedback loops.

It's worth noting that the Qwen3.5 series is currently in a "buggy, unsupported" state, while Qwen3.6 is still being implemented. Both Qwen3.5 and Qwen3.6 require separate adaptation of the draft model's token acceptance logic due to architectural changes (such as adjustments to attention mechanisms or positional encodings), which explains the delayed support. Given that these two versions are currently the most widely used in the community, their full support will be the truly critical milestone. Once the relevant features are merged and live, they deserve a dedicated round of benchmarking.

Performance: Significant Single-User Experience Gains

The vLLM team published a performance stress-test curve for this algorithm. The test environment was DeepSeek V4 (D300 machines), using a typical long-context scenario of 8K input / 1K output.

How to read the chart:

• X-axis: Average token rate per user (tokens per second) — higher values mean better single-user experience;
• Y-axis: Overall throughput per GPU per second — higher values mean better cluster efficiency;
• Blue line (Baseline): Performance without speculative decoding;
• Green line (DSpark): Performance with speculative decoding enabled.

Curves further toward the upper right indicate better performance.

Single-User Token Rate: Nearly 150× Improvement

The biggest gain from speculative decoding shows up in per-user generation speed. On the X-axis dimension, DSpark shows a nearly 150× gap over the Baseline. This means users in low-concurrency, single-user interaction scenarios will noticeably experience faster token output — which is precisely the core advantage of speculative decoding.

Understanding the technical basis for this remarkable improvement requires the concept of TPOT (Time Per Output Token), which is the primary metric speculative decoding optimizes. In low-concurrency scenarios, the GPU's Tensor Core compute units have significant idle capacity (the system is Memory-bound rather than Compute-bound). Speculative decoding converts this idle compute into parallel draft verification at near-zero marginal cost, which is why single-user speed gains are so dramatic.

The principle is simple: a small "draft model" quickly generates several candidate tokens, and the large model then verifies them all in a single parallel forward pass, "confirming" multiple tokens per forward computation. When concurrency is low and GPU compute is plentiful, this strategy of "trading compute for lower latency" yields its highest returns.

Overall Throughput: ~40%–50% Improvement

By comparison, the improvement on the Y-axis (overall throughput) is more moderate — based on the curve comparison, total throughput increases by approximately 40%–50%.

This difference is consistent with the fundamental characteristics of speculative decoding: when the system is under high concurrency and the GPU is saturated, compute headroom shrinks. Each parallel verification pass must compete with other requests for compute resources, diluting the advantage of "verifying multiple tokens in parallel." The throughput-level gains are far smaller than the per-user latency gains. In other words, this technology is more oriented toward optimizing "user experience" than "peak throughput" — it is best suited for latency-sensitive, moderate-concurrency interactive scenarios like consumer-facing chat and code completion, rather than high-QPS batch processing workloads.

Significance for the Open-Source Ecosystem

Looking at the longer-term trend, as DSpark continues to mature within vLLM's main branch, virtually all mainstream open-source models stand to benefit from this acceleration. For production deployment of open-source models, this is a strongly positive signal:

1. Lower barrier to entry: No need to implement complex speculative decoding logic from scratch — updating the image version is all it takes to get the speedup;
2. Tangible experience gains: The large improvement in single-user token rate directly enhances real-time responsiveness for interactive applications like chat and code completion;
3. Accelerating mainstream adoption: Merging into the main branch means faster iteration cycles, broader community testing, and more stable long-term maintenance.

That said, there are still gaps to fill — support for the community's most heavily used versions, Qwen3.5 and Qwen3.6, is not yet ready. Teams planning to use speculative decoding in production should closely monitor upcoming vLLM releases, run their own benchmarks once their target models are officially supported, and then decide whether to roll it out.

Summary

The merger of DeepSeek's speculative decoding algorithm into vLLM's main branch represents a substantive advance in open-source inference acceleration. Its underlying mechanism — using a draft model's low-cost speculative generation to enable the main model's batched parallel verification — makes full use of idle GPU compute, delivering a nearly 100× improvement in single-user token speed and a ~40%–50% gain in overall throughput. These characteristics make it especially well-suited for applications where real-time interactive experience matters most. As support expands to high-frequency models like Qwen3.5 and Qwen3.6, speculative decoding is poised to become a standard acceleration technique for open-source LLM deployment.

Key Takeaways