DeepSeek Open-Sources DeepSpec: A Full-Stack Speculative Decoding Library with 60–85% Inference Speedup

DeepSeek Drops a Major Release: Full-Stack Speculative Decoding Library Now Open Source

The center of gravity in the large model race is quietly shifting from training efficiency to inference acceleration. DeepSpec, the speculative decoding library open-sourced by the DeepSeek team, lands squarely at this inflection point.

Let's start with three hard numbers from production environments:

• Real-world testing on DeepSeek V4 in production: inference response speed improved by 60% to 85%
• Engine throughput improved by 51%
• Throughput under high-concurrency scenarios surged by 661%

These aren't lab-generated theoretical figures — they were measured on clusters running tens of thousands of GPUs. For any team deploying large models at scale, this kind of speedup translates directly into cost savings and a better user experience.

Industry Context: Inference costs have become a core bottleneck limiting the at-scale deployment of large models. According to a 2024 Andreessen Horowitz research report, inference costs account for 60%–80% of total cloud computing spending at leading AI application companies, and in some vertical domains, inference costs even exceed revenue. Against this backdrop, the commercial value of inference acceleration far outweighs gains in raw model capability — a 60%–85% speed improvement directly corresponds to cutting in half the compute needed to serve the same number of users. This explains why OpenAI, Anthropic, Google, and other top-tier organizations are all investing heavily in inference infrastructure optimization.

What Is DeepSpec

DeepSpec is a full-stack speculative decoding library open-sourced by the DeepSeek team, covering training, evaluation, and deployment in a single package.

It integrates three major components:

• Three mainstream Draft Model algorithms: covering different deployment scenarios
• Nine pre-trained Hugging Face Checkpoints: ready to use out of the box, no training from scratch required
• Nine evaluation benchmarks: spanning math, code, and conversational tasks

The Core Mechanism of Speculative Decoding

The idea behind speculative decoding is intuitive: a small model drafts tokens first, then the large model verifies them in batch.

Think of it like an intern drafting a document and a manager reviewing it all at once. Crucially, this process is mathematically lossless — the final output is identical to what the large model would generate on its own, just much faster.

Technical Background: Speculative Decoding was formally introduced by the Google Brain team in a 2022 paper of the same name, followed by research from DeepMind, Meta, and others. The core insight is that the inference bottleneck in autoregressive large models is not compute, but memory bandwidth — every token generation requires loading tens of billions of parameters, causing severe GPU underutilization. Speculative decoding addresses this by introducing a "draft model" that is 1–2 orders of magnitude smaller to generate multiple candidate token sequences in advance, which the target large model then verifies in parallel. This transforms serial token-by-token generation into batch verification, fully exploiting the GPU's parallel compute capacity. Mathematically, the method uses rejection sampling to guarantee that the output distribution is identical to the original large model — making it a truly lossless acceleration technique, fundamentally different from lossy compression approaches like quantization or distillation.

What DeepSpec does is standardize and productionize this previously fragmented process, so teams don't have to reinvent the wheel.

Three Algorithms, One Unified Library

DeepSpec ships with three representative speculative decoding algorithms, covering major use cases from server-side to local deployment:

• DeepSeek's proprietary algorithm: built around a confidence-based scheduling mechanism, designed for high-concurrency scenarios to maintain stability under load
• DeepFlash: a fast diffusion approach under the MIT license, ideal for single-GPU local deployment
• EAGLE-3: a multi-candidate exploration method from the SGLang team, suited for scenarios demanding maximum token acceptance rates

Algorithm Technical Details: The three algorithms represent different technical approaches within the speculative decoding space. DeepSeek's confidence-based scheduling mechanism addresses the instability of draft model acceptance rates under high concurrency — dynamically adjusting draft length as system load increases to strike the optimal balance between throughput and acceptance rate. DeepFlash is based on a fast diffusion approach that accelerates generation in single-GPU local environments by decoding multiple token positions in parallel, making it especially suitable for memory-constrained edge deployments. EAGLE-3 (Extrapolation Algorithm for Greater Language-model Efficiency) is the SGLang team's third-generation autoregressive head method. By making draft predictions in the hidden feature space rather than the vocabulary space, it significantly improves token acceptance rates — particularly on tasks with concentrated distributions like code and mathematics.

Each algorithm has its strengths, and users can choose based on their specific needs: online serving, local inference, or research reproduction.

Up and Running in Five Minutes: Three Steps

DeepSpec puts a strong emphasis on ease of use. The official quickstart requires just three steps:

1. git clone the repository
2. Download pre-trained checkpoints via the Hugging Face CLI
3. Launch the service with vllm serve

About vLLM: vLLM is a high-performance large model inference engine open-sourced by UC Berkeley in 2023. Its core innovation, PagedAttention, draws from operating system virtual memory paging to fundamentally solve the KV Cache fragmentation problem, boosting GPU memory utilization from the original 20%–40% to near 100%. vLLM has become the de facto standard for production inference deployment, with OpenAI-compatible API support and widespread adoption by providers like Anyscale and Together AI. DeepSpec embeds speculative decoding into vLLM's scheduling system via its plugin mechanism, allowing users to enable acceleration purely through vllm serve configuration parameters — no application-layer code changes required, dramatically lowering the integration barrier.

Real-world results are equally impressive: on four H100 GPUs, Qwen3-8B's serving concurrency jumped from 30 to 60–80 concurrent requests — nearly doubling — achieved through the combined effect of vLLM's continuous batching and speculative decoding.

What the Nine Checkpoints Cover

The nine Hugging Face Checkpoints follow a four target models × three algorithms grid:

• Target models include Qwen3 at 4B, 8B, and 14B, plus Gemma 4 at 12B
• Each paired with all three algorithms

This covers virtually all of the most commonly used open-source base models today. The experimental data in Table 1 of the paper also uses this exact configuration, making it easy for researchers to align with and reproduce results.

Nine Evaluation Benchmarks: Full-Spectrum Coverage

Acceleration alone isn't enough — the results must be quantifiable. DeepSpec ships with nine evaluation benchmarks spanning three capability dimensions:

• Mathematical reasoning: GSM8K, MATH-500, etc. — testing acceleration under complex reasoning
• Code generation: HumanEval, MBPP, LiveCodeBench, etc. — validating acceptance rates in code scenarios
• General conversation: MT-Bench, Alpaca, Arena-Hard v2, etc. — covering real-world dialogue distributions

Why These Benchmarks Matter: This tiered design addresses the long-standing "cherry-picking" problem in inference acceleration research. GSM8K and MATH-500 represent high-reasoning-chain scenarios where token distributions are highly concentrated and speculative decoding acceptance rates are typically highest. HumanEval and MBPP represent code completion, where syntactic constraints make draft predictions more accurate. Arena-Hard v2 and MT-Bench represent real user dialogue distributions with the highest token diversity — the closest to a true production stress test. Many prior acceleration methods only reported speedups on concentrated-distribution tasks, where gains in general conversation scenarios often fell short. A comprehensive report across nine benchmarks gives researchers and engineers a genuine picture of algorithm performance across different business contexts, rather than being misled by average numbers.

The real value of this benchmark suite is making the speedup ratios of different algorithms across different tasks clearly visible — avoiding the problem of "average acceleration" masking scenario-specific variation.

Four Real-World Use Cases

DeepSpec's true value lies in its ability to solve practical problems. The team highlights four representative deployment scenarios:

Scenario 1: Self-hosted inference services — cut costs. Combined with vLLM, per-inference costs can be cut roughly in half — a significant saving for continuously running services.

Scenario 2: Agent and RAG applications — improve experience. Speculative decoding compresses response times to roughly one-third, dramatically reducing interaction latency. This is especially critical for interaction-intensive Agent applications, which typically involve multiple rounds of tool calls and model inference interleaved together — each inference latency compounds directly into user wait time, making the speedup gains particularly impactful here.

Scenario 3: Teams training their own models — accelerate integration. With DeepSpec's training pipeline, a team can complete speculative decoding adaptation for their own model within two weeks, significantly shortening engineering timelines.

Scenario 4: Researchers — easy reproduction. All three algorithms and nine benchmarks can be pulled and run directly to fully reproduce the results in Table 1 of the paper — highly accessible for academic use.

Why This Matters

Taken together, DeepSpec stands out on four dimensions:

• Solves real problems: directly addresses inference cost and latency — the two core pain points of production deployment
• Full-stack open source: training code, nine checkpoints, nine evaluation benchmarks, and vLLM integration are all included
• Academically friendly: paper and experiments are fully reproducible
• Commercially friendly: MIT license — free to use in commercial products with no restrictions

The project is now live on GitHub (DeepSeek-AI/DeepSpec) and garnered 6,729 stars at launch. The accompanying paper (arXiv: 2607.05147) and pre-trained checkpoints are all publicly available.

DeepSpec's strategic choice of the MIT license carries broader significance: on one hand, it rapidly builds an ecosystem and helps push speculative decoding toward becoming an industry standard; on the other, by lowering the inference cost barrier across the industry, it indirectly expands the application frontier for large models. For small and mid-sized AI teams, libraries like DeepSpec mean they can achieve near-top-tier deployment efficiency without staffing a dedicated inference optimization team.

As large model capabilities mature and competitive focus shifts toward engineering efficiency, open-source tooling like DeepSpec — squarely focused on the inference layer — is becoming one of the most important categories for infrastructure teams to watch. What it lowers is not just the technical barrier, but the cost curve for the entire industry to deploy large models at scale.