DeepSeek Open-Sources DeepSpec: A Full-Stack Breakdown of Speculative Decoding

DeepSeek has made another move, open-sourcing a project called DeepSpec. Let's clarify one thing upfront: this isn't a new model. It's a complete training and evaluation codebase built around Speculative Decoding — one of the most mainstream techniques for accelerating LLM inference. DeepSpec open-sources the entire pipeline in one shot, from data preparation and draft model training to speed benchmarking, and ships with three draft model algorithms and 12 pre-trained checkpoints.

This article walks through everything DeepSpec actually releases, from technical principles to engineering details.

What Problem Does Speculative Decoding Solve?

To understand DeepSpec's value, you first need to understand the pain point speculative decoding addresses. Large language models generate text one token at a time, and each token requires a full forward pass through the model. The problem is that modern GPUs are enormously powerful — generating a single token barely scratches the surface of their compute capacity, leaving most of it idle while users wait. That's the root cause of slow inference.

In engineering terms, this is called the "Memory-Bound" problem — a structural imbalance between compute (FLOPS) and memory bandwidth in modern GPU architectures. Take the NVIDIA H100: its peak compute exceeds 3,000 TFLOPS, but HBM3 memory bandwidth is around 3.35 TB/s. During autoregressive generation, producing each token requires loading tens of billions of parameters from memory to compute units. This process is bottlenecked by bandwidth, not compute, leaving effective GPU utilization often below 1% of theoretical peak. The core value of speculative decoding is to increase arithmetic intensity per forward pass by verifying multiple candidate tokens in batch — directly addressing this structural bottleneck.

The key insight behind speculative decoding is elegant: bring in a small draft model to quickly guess the next several tokens (say, 5 at once). Because the draft model is lightweight, this happens very fast. Then feed all those guesses to the large target model at once — it can verify all of them in a single forward pass. Consecutive correct guesses are accepted; the first wrong one and everything after it is discarded, and generation resumes from that point.

The crucial property here: the target model only verifies and approves. Every token in the final output is one the target model endorses. This "lossless" guarantee comes from the mathematics of rejection sampling — as long as the verification step follows a specific accept-reject rule, the final output distribution is statistically equivalent to what the target model would produce on its own. Specifically, when a draft token's probability under the target model exceeds its probability under the draft model, it's accepted unconditionally; otherwise it's accepted with probability equal to their ratio. This rule was formally proven in Google DeepMind's 2022 paper "Accelerating Large Language Model Decoding with Speculative Sampling." This distinguishes speculative decoding from lossy compression approaches like quantization or pruning — it's a zero-quality-compromise acceleration path. Even when the draft model guesses wrong, the final output is identical to what the large model would have generated token by token. In technical terms: byte-level identical, lossless.

The key metric for measuring how well this works is the acceptance rate: the proportion of draft tokens that the target model accepts. The higher the acceptance rate, the more tokens you get per forward pass, and the greater the speedup. This means the quality of speculative decoding is almost entirely determined by how good the draft model is.

Three Draft Model Algorithms: Ego3, DeepFlash, and DeepSpark

DeepSpec includes three draft model algorithms, each representing a different technical approach.

Ego3: The Autoregressive Approach

Ego3 is the most widely used approach — a standard token-by-token generation scheme, just much lighter than the target model. DeepSpec directly reuses the implementation from the open-source framework SpecForge. SpecForge is a significant framework in the speculative decoding ecosystem, providing standardized data formats, training pipelines, and evaluation interfaces. Its existence signals that speculative decoding has moved from academic papers into systems engineering, with a dedicated toolchain ecosystem forming around it.

DeepFlash: The Parallel Diffusion Approach

The second approach is more aggressive. DeepFlash predicts an entire block of tokens simultaneously in a single forward pass, independent of order. This borrows from diffusion models in image generation and BERT-style masked language modeling — just as a diffusion model can update all pixel positions in parallel during a single denoising step, DeepFlash uses parallel prediction heads to output candidates for all token positions within a window simultaneously, eliminating the serial dependency of autoregressive draft generation. The advantage is extreme speed — one forward pass covers an entire block. The tradeoff is that each position can't leverage information from other positions in the same window, so prediction quality degrades progressively for later tokens. This is known as suffix decay — the further out you go, the higher the error rate.

DeepSpark: DeepSeek's Core Approach, Co-Developed with Peking University

DeepSpark is DeepSeek's own solution and the centerpiece of this release, developed in collaboration with Peking University. Its goal is to capture the advantages of both previous approaches.

DeepSpark's draft model uses a semi-autoregressive architecture: a parallel backbone (like DeepFlash, predicting an entire block at once for speed) combined with a lightweight autoregressive refinement head (like Ego3, correcting predictions token by token) to suppress suffix decay. The front portion of a block gets DeepFlash's speed; the back portion gets Ego3's accuracy.

DeepSpark's second key innovation is reframing speculation as a scheduling problem. Previous speculative decoding approaches sent all draft tokens to the target model for verification regardless of quality. DeepSpark adds an adaptive scheduler that decides whether a batch of candidates is worth verifying and how deep to verify — high-quality drafts get more verification, low-quality ones are skipped, conserving compute.

In offline testing, comparing only the draft models themselves (scheduler disabled), DeepSpark's acceptance length is 27–31% higher than Ego3 and 16–18% higher than DeepFlash.

A Fully Reproducible Training Pipeline

The algorithm is only part of what DeepSpark offers. Its real value is transforming speculative decoding from scattered engineering tricks into a reproducible training pipeline. The full pipeline has three stages.

Stage 1: Data Preparation

Download training prompts, regenerate responses using the target model, and organize everything into a target cache for draft model training. There's a real practical barrier here: the default setup targets Qwen3's 4-billion-parameter model, and that cache requires approximately 38TB of storage — not something casual practitioners can easily provision.

This storage requirement stems from the inherent demands of knowledge distillation training. Training a draft model using only hard text labels creates a systematic divergence between the draft model's output distribution and the target model's distribution, which reduces acceptance rates. The standard approach is to save the target model's full probability distributions (logits) or temperature-softened soft labels at each token position as the training signal. For Qwen3-4B with a vocabulary of roughly 151,936 tokens, each position requires storing ~150K float32 values (~600KB). Multiplied across tens of millions of training samples, total storage rapidly balloons to tens of terabytes. This cost structure isn't specific to DeepSpec — it's a shared constraint of knowledge distillation-based training approaches, and it means this pipeline is primarily aimed at teams with data-center-scale storage.

Stage 2: Training

The code assumes a single node with 8 GPUs by default — one worker per GPU, launched with a single command. The config file specifies the algorithm and target model, for example DeepSpark targeting Qwen3-4B.

Stage 3: Evaluation

Once the draft model is trained, a benchmark suite measures acceptance rates and real-world speedup. DeepSpec includes 9 evaluation sets with broad coverage: math benchmarks (GSM8K, MATH500), code benchmarks (HumanEval, MBPP, LiveCodeBench), and dialog/general benchmarks (MTBench, AlpacaEval, ArenaHard v2) — all widely recognized standards in the open-source community.

12 Open-Source Checkpoints and Usage Constraints

Along with the code, DeepSpec releases 12 pre-trained draft model checkpoints, all hosted on Hugging Face and available for direct download. These 12 checkpoints represent three algorithms × four target models: Qwen3 at 4B, 8B, and 14B parameters, plus Google's Gemma 4 at 12B.

One critical point to be aware of: all checkpoints were trained in non-thinking mode.

If your target model runs in thinking mode (producing a chain-of-thought reasoning trace before the final answer), the official recommendation is to fine-tune a draft model for your specific use case — using these checkpoints directly will yield degraded performance.

A few other constraints worth knowing upfront:

• Extremely high storage requirements: The default target cache requires 38TB — this is primarily aimed at teams with substantial compute infrastructure.
• Training requires at least 8 GPUs: The code is written for single-node 8-card setups. This is an industrial-scale reproducibility tool, not a plug-and-play inference acceleration plugin.
• Thinking mode requires separate adaptation: The open-source checkpoints are all trained for non-thinking mode; thinking-mode models need to be fine-tuned separately.
• Only covers Qwen3 and Gemma: Using other models — including DeepSeek's own V4 — requires training a new draft model from scratch using this pipeline. It provides methodology and tooling, not a universal acceleration package.

Production Performance and Boundary Conditions

The DeepSpark algorithm is already deployed in DeepSeek's V4 model production service. On V4 Flash, compared to the MTP-1 baseline (the most conservative configuration of DeepSeek's in-house Multi-Token Prediction mechanism), single-user generation speed improves by 60–85%; on the larger V4 Pro, the improvement is 57–78%.

Understanding these numbers requires some context around MTP. MTP (Multi-Token Prediction) is DeepSeek's proprietary acceleration mechanism introduced during V-series model training — by adding multi-token prediction auxiliary objectives during pretraining and supervised fine-tuning, the model backbone can output candidates for multiple subsequent tokens at inference time without needing a separate draft model. MTP-1 is the most conservative configuration, predicting only one additional token per step and adding roughly 0.8B parameters in prediction heads. DeepSpark, as an independently trained external draft model, derives its predictive capability from specialized alignment training rather than modifications to the backbone architecture. The two target different bottlenecks and can be deployed together in V4 production: MTP provides baseline acceleration without additional training, and DeepSpark further improves acceptance rates on top of that, making them complementary rather than competing. DeepSpark's additional 60–85% speedup on top of the MTP-1 baseline validates this complementary effect in practice.

The qualifiers here matter: relative to the MTP-1 baseline, in single-user scenarios, with byte-level lossless output, and with zero retraining of the target model.

It's also worth emphasizing: the 60–85% figures come from DeepSpark running in DeepSeek's own V4 production environment. The 12 open-source checkpoints are demonstration models for Qwen3 and Gemma, intended for community reproduction and further development. They share the same algorithmic core but serve different contexts — the numbers don't transfer directly. This is where the codebase shows its rigor: it keeps these distinctions clear throughout.

Why This Open-Source Release Matters

For developers and researchers, the most significant aspect of DeepSpec is that it pulls a technique previously hidden in the backends of major providers completely into the open. Before this, speculative decoding was more like a secret optimization running silently on inference servers — how well it worked and how to tune it was largely opaque to the outside world.

DeepSpec releases the full three-stage pipeline — data preparation, training, and evaluation code — along with 12 directly downloadable checkpoints, all under the MIT License, allowing commercial use. The MIT License is one of the oldest and most widely adopted permissive open-source licenses. Unlike the GPL (which requires derivative works to also be open-sourced) or Apache 2.0 (which includes explicit patent grant clauses), the MIT License only requires preserving the copyright notice. It places no restrictions on how derivatives are distributed, commercialized, or whether source code must be disclosed. This means companies can integrate DeepSpec code directly into proprietary commercial inference services without publishing internal modifications or seeking additional authorization from DeepSeek — a licensing choice that significantly lowers the legal barrier for industrial adoption and is an important prerequisite for DeepSpec reaching production environments. For teams focused on model deployment and inference optimization, this is an engineering reference they can immediately adapt. Teams that previously had no choice but to call someone else's API can now use this toolchain to train custom draft models for their own models, autonomously reducing inference costs.

DeepSpec's acknowledgments also make clear that the training framework builds on SpecForge, the Ego3 implementation comes from SpecForge, DeepFlash's design comes from the DeepFlash project, and the target models used are Qwen3 and Gemma — it's an integration built on the shoulders of multiple open-source projects.

Inference acceleration is no longer an exclusive trade secret of large tech companies. DeepSpec makes that statement concrete.