DeepSeek DSpec Hands-On: How Speculative Decoding Accelerates LLM Inference by 6x

Introduction: Another Open-Source Gem from DeepSeek

DeepSeek recently open-sourced an inference acceleration toolkit called DSpec (also referred to as DeepSpec in the video), which incorporates its latest speculative decoding technology (the DSPI/MTP series). This isn't just a simple demo—it's a complete GitHub repository containing model code, training scripts, and evaluation tools. Anyone can download it, verify it, and even reproduce the performance figures published in DeepSeek's paper themselves.

A Bilibili content creator (reposting from the YouTube channel Fahd Mirza) ran a hands-on test on a machine equipped with an RTX A6000 (48GB of VRAM), attaching DSpec's 1-billion-parameter draft model to Qwen3's 3.4B target model. The results were quite impressive: on structured tasks, the large model could accept up to 6 tokens in a single pass, theoretically compressing the number of costly large-model computations to roughly one-sixth. This article breaks down how DSpec works and its real-world performance based on this hands-on test.

About the test hardware: The RTX A6000 is NVIDIA's flagship GPU aimed at the professional workstation market. Based on the Ampere architecture, it packs 10,496 CUDA cores and 48GB of GDDR6 memory (768 GB/s bandwidth). Its ECC error-correcting memory support makes it well-suited for professional workloads requiring long-term, stable operation. By comparison, the consumer-grade RTX 4090 offers higher FP32 compute (82.6 TFLOPS versus the A6000's 38.7 TFLOPS), but its VRAM capacity is only 24GB. Notably, in this test the two models combined occupied only about 12GB of VRAM—less than a quarter of the A6000's capacity. This means DSpec's collaborative approach is fully viable on consumer GPUs (such as the RTX 3090 or RTX 4090), and even some 16GB mid-range GPUs (like the RTX 4080) are theoretically within range, dramatically lowering the deployment barrier for everyday developers.

DSpec Core Principles: Letting a Small Model "Draft" for the Large Model

The Basic Logic of Speculative Decoding

To understand DSpec, you first need to understand the speculative decoding it relies on.

Why is autoregressive generation so slow? Today's mainstream large language models use the autoregressive paradigm: each time, the model reads all previously generated tokens, predicts the single most likely next token, appends it to the end of the sequence, and repeats the process. This mechanism has inherent serial dependency—the Nth token must wait until the (N-1)th token is fully generated. For large-scale models, generating each token requires a full traversal of all parameters, severely wasting the GPU's powerful parallel computing capabilities. This is precisely why even top-tier GPU clusters typically produce large-model output at only a few dozen tokens per second.

The academic evolution of speculative decoding: Speculative decoding didn't appear out of nowhere—it has a clear academic lineage. In 2023, Yaniv Leviathan and colleagues at Google Brain and a DeepMind team almost simultaneously and independently published research, jointly laying the theoretical foundation for this paradigm. Its core mathematical guarantee is this: as long as the large model's (the target model's) accept-reject sampling mechanism is correctly designed, the final output token distribution is exactly equivalent to the original autoregressive generation—this is the strict mathematical meaning of "lossless" acceleration. Since then, institutions like Meta and Hugging Face have successively proposed variants such as Medusa (multi-head parallel drafting), EAGLE (speculating at the feature layer rather than the word-embedding layer to further improve draft quality), and Lookahead Decoding (a parameter-free approach based on Jacobi iteration that requires no additional draft model). Speculative decoding has gradually evolved into an active subfield of inference optimization.

The core insight of speculative decoding is this: the compute cost for a GPU to verify a batch of tokens is close to the cost of verifying a single token—because the parallel computing nature makes batch verification almost "free." The idea is to have a small, fast draft model guess several tokens in advance, then have the large model verify all these guesses in a single batch pass. Essentially, this brings the GPU's parallel advantage from the training phase into the inference phase.

DeepSeek's earlier base version of MTP (Multi-Token Prediction) guessed only one token at a time, and its draft model was not an independently trained external component but was co-optimized during the main model's training—this naturally aligned the draft model's output distribution with the target model, fundamentally raising the ceiling on the acceptance rate. This is also the core difference between DSpec and post-hoc distillation approaches. DSpec builds on this with two key upgrades.

DSpec's Two Major Improvements

First, making drafts more coherent. By introducing a tiny memory head, DSpec prevents the draft model from "breaking down midway" during consecutive predictions, maintaining semantic coherence across multiple guessed steps. The memory head is essentially a lightweight recurrent state mechanism that compresses and encodes the hidden state of prior draft tokens before passing it to the next prediction step. Compared to a full attention mechanism, it has an extremely small parameter count and adds almost no inference latency. Compared to traditional recurrent networks like LSTM or GRU, the memory head retains only the small number of state dimensions crucial to draft coherence, striking a delicate balance between computational efficiency and semantic preservation.

Second, making verification smarter. DSpec introduces a dynamic scheduler that only selects guesses "worth checking" for verification: verify more when the system is idle, verify less when the system is busy—the more accurate the guesses, the less wasted effort. The final output is entirely consistent with token-by-token generation, achieving lossless acceleration.

The Full Hands-On Process: From Cloning the Repo to Running Benchmarks

Hidden Pitfalls in Setting Up the Environment

The test workflow is fairly standard: clone the GitHub repo, create a virtual environment, and install dependencies. But the creator specifically pointed out an easy-to-hit pitfall—the evaluation script relies on the PrettyTable library to print the results table, but DSpec forgot to include it in the dependency list. The script crashes at the very last step after completing all computations, so you need to manually run pip install prettytable, or all your effort goes to waste.

Model Downloads and VRAM Usage

The test downloaded two models: Qwen3's 3.4B target model at about 8GB (roughly one minute to download), and the DSpec draft model at only about 2.8GB. When running the core script eval.py, both models load simultaneously and run collaboratively in speculative decoding mode, reporting the draft model's guess acceptance rate in real time.

The VRAM performance is also noteworthy: initial loading occupied about 11GB, and throughout inference it stayed stably below 12GB—quite restrained for a dual-model collaborative configuration. This data also confirms the maturity of quantization techniques (INT4/INT8) and modern VRAM management strategies: two models with a combined parameter count of over 4B actually used far less VRAM than their theoretical value under FP16 precision. INT4 quantization compresses each parameter from 16 bits to 4 bits, theoretically reducing VRAM usage to one-quarter of the original at the cost of a slight precision loss; modern quantization frameworks like bitsandbytes and GPTQ have tuned this trade-off to a level that is nearly imperceptible for most tasks.

Reproducing the Official Data: The True Value of Open Source

Hands-On Results on GSM8K and MTBench

For demonstration purposes, the creator trimmed the full evaluation—originally covering 9 benchmarks and requiring several hours—down to two tasks with 20 samples each. The choice of these two benchmarks is quite representative:

GSM8K (Grade School Math 8K) was released in 2021 by OpenAI researcher Karl Cobbe and colleagues, with the original intent of filling a gap in NLP benchmarks at the time regarding "real-world problems requiring multi-step reasoning." The dataset contains 8,500 math word problems requiring multi-step reasoning. A key design decision was requiring the final answer to appear in numeric form, which makes automated evaluation possible and also makes it a natural yardstick for measuring structured generation capability. From an information-theoretic perspective, the output entropy of math problems is extremely low—given the problem and reasoning process, the conditional probability distribution of the next token is highly concentrated, which is precisely why the DSpec draft model achieves an extremely high consecutive hit rate on this task. Each problem requires an average of 2 to 8 reasoning steps, making it a standard test for measuring a model's structured generation capability. Notably, as large-model capabilities have rapidly improved, GSM8K is now close to saturation for top-tier models (accuracy exceeding 95%), and academia is shifting to harder datasets like MATH and AIME to differentiate model capability boundaries. However, GSM8K's status as a representative task for "low-entropy structured output" in inference acceleration research remains solid.

MTBench (Multi-Turn Benchmark) was proposed in 2023 by the UC Berkeley LMSYS team (also the creators of Vicuna and Chatbot Arena). Its core innovation was introducing GPT-4 as a judge (LLM-as-a-judge), converting subjective conversation quality assessment into quantifiable scores and pioneering the new paradigm of using large models to evaluate large models. MTBench covers 8 open domains including writing, role-play, and reasoning, deliberately choosing scenarios with highly diverse outputs to examine a model's multi-turn instruction-following ability. It's worth noting that the LLM-as-a-judge paradigm itself is controversial: GPT-4 as a judge may assign higher scores to outputs stylistically similar to its own, and this potential bias (position bias, verbosity bias) remains a methodological issue under ongoing academic discussion. The combination of the two forms a stark contrast: one represents highly predictable structured output, the other represents open-ended creation full of variables—essentially, this is a quantitative manifestation of "predictability" versus "creativity" within the speculative decoding framework.

The results closely matched the official data:

• GSM8K: measured acceptance length of 6, versus DeepSeek's paper value of 6.1;
• MTBench: measured acceptance length of 3.65, versus 3.64 in the paper under the same settings.

On a single GPU, in under a minute, the benchmark data officially published by DeepSeek was reproduced. As the creator put it: "This isn't marketing hype—you can verify it yourself." This is precisely the core value proposition of an open-source toolkit.

What an Acceptance Length of 6 Means

An acceptance length of 6 means that in each round of inference, the large model accepted 6 tokens from the draft model in a single pass, rather than generating token by token in the traditional manner. Mathematically: if the draft model proposes K tokens, the large model verifies them one by one and rejects at position i, then this round effectively generates i+1 tokens (the i accepted draft tokens plus the correction token the large model generates itself), while the large model performed only a single forward pass. The number of costly large-model computations thus drops to roughly one-sixth—the direct source of the inference speedup.

It's worth noting that actual end-to-end throughput improvements typically fall in the 2x to 4x range, rather than the 6x that an acceptance length of 6 might theoretically suggest. The gap stems from multiple engineering factors: the overhead of KV cache (Key-Value Cache) management is the most critical among them. The KV cache is a core acceleration mechanism in Transformer inference; it caches the attention key-value pairs corresponding to already-generated tokens to avoid recomputation. But in speculative decoding, when draft tokens are partially rejected by the large model, the KV cache needs to "roll back" to the state of the last accepted token, introducing additional memory management complexity. An efficient implementation requires support for the Tree Attention mechanism—allowing the large model to verify multiple parallel draft branches simultaneously within a single forward pass, rather than processing them serially, thereby maximizing GPU utilization. The engineering implementation of tree attention requires custom CUDA kernels to handle irregular attention masks, which is precisely the technical barrier that inference frameworks like vLLM and TensorRT-LLM compete to support speculative decoding. KV cache rollback, GPU memory bandwidth bottlenecks, and batch scheduling together constitute the "efficiency tax" between theory and practice, making the speedup ratio a nonlinear function of acceptance length, with the specific value also depending on hardware configuration and task type.

Diving into the Details: The Decay Pattern of Acceptance Rates

Why Math Tasks See More Significant Acceleration Than Chat Tasks

GSM8K (structured math) reached an acceptance length of 6, while MTBench (open chat) reached only 3.65. The underlying reason is that chat content is harder to predict. From an information-theoretic perspective, this is equivalent to the difference in conditional entropy between the two task types: for math problems, given the context, the conditional probability distribution of the next token is highly concentrated (low entropy), and the draft model has a higher probability of guessing correctly in succession; open-ended dialogue is full of uncertainty, with a flat conditional probability distribution (high entropy), so guesses quickly "go off track."

Quantifying this difference in the language of Shannon entropy: if the highest-probability token at a given position has a probability of 0.9 (the math scenario), then the entropy at that position is about 0.47 bits; if the probability is uniformly distributed across 10 tokens (the chat scenario), then the entropy is about 3.32 bits—a difference of nearly 7x, which directly manifests as the difference in the draft model's acceptance rates across different tasks. This also reveals a universal applicability rule for speculative decoding: the lower a task's output entropy (i.e., the more predictable the answer), the more significant the acceleration benefit of speculative decoding. Highly structured tasks like code generation and SQL queries are typically where speculative decoding benefits the most, while open tasks like free-verse poetry and brainstorming see relatively limited gains.

Acceptance Rate Decays Progressively with Position

The draft model doesn't just guess one token—it produces a candidate sequence of about 7 tokens at once. The acceptance rate at position 0 represents the frequency of the first guess being correct, position 1 represents the second, and so on. The test clearly shows the decay pattern:

• GSM8K: the acceptance rate gradually declines from 0.92 at position 0 to about 0.53;
• MTBench: the acceptance rate plummets from 0.78 to 0.13.

This decay phenomenon has a rigorous probabilistic explanation. Let the draft model's independent acceptance probability at each position be p; then the joint acceptance probability at position k is approximately of the form p^k. Taking GSM8K as an example, an acceptance rate of 0.92 at position 0 means that if the positions were independent, the acceptance rate at position 5 would theoretically be about 0.92^5 ≈ 0.66, whereas the measured value is about 0.53. The difference stems from the autocorrelation of errors—deviations in earlier guesses systematically contaminate the context of subsequent predictions. This error propagation mechanism is highly analogous to state transitions in a Hidden Markov Model: the hidden state at each step (the draft model's internal representation of the context) depends on the previous step, and once an error is introduced, it is amplified through the conditional probability chain, causing the measured decay rate to be faster than the theoretical prediction under the independence assumption. Intuitively: the first guess is based on the real large-model context and has a high accuracy rate; by the sixth guess, the model is guessing "on top of its own previous guesses," and errors at each step accumulate and distort the input distribution of the next step.

The Significance of DSpec's Dynamic Scheduling

This decay pattern is precisely where DSpec's core value lies. A naive system would verify all 7 candidate tokens, wasting compute on those "doomed-to-fail" later positions. DSpec, through confidence scores and calibration tables, dynamically decides the actual number of tokens that need to be verified—verifying more when confidence is high, and stopping early when confidence is low.

This is essentially the application of the Optimal Stopping problem in engineering practice—a classic problem framework in operations research and probability theory, whose most famous instance is the "Secretary Problem": when sequentially observing a series of candidates, deciding when to stop and hire the current best candidate to maximize the probability of selecting the globally optimal one. Mathematically, solutions to optimal stopping problems typically involve the Bellman equation—using dynamic programming to compare the expected reward of the "current decision" against the expected reward of "continuing to wait," making an optimal choice at each step. DSpec's scheduler estimates in real time the ratio of the expected reward of continuing to verify the next draft token (acceptance probability × compute saved) to the verification cost, and decisively terminates the verification chain when the ratio falls below a dynamic threshold. For open tasks like MTBench, the system proactively terminates the verification chain before the acceptance rate plummets, allocating the saved compute to more valuable tasks; for structured tasks like GSM8K, it fully exploits the high-acceptance-rate window, extending the verification chain to its longest. This adaptive resource allocation strategy is highly isomorphic in mathematical structure to optimal execution algorithms in finance and early-termination strategies in search engines, embodying the universality of statistical decision theory in system optimization, and is a key advance that sets DSpec apart from traditional speculative decoding.

Conclusion

The open-sourcing and hands-on testing of DSpec bring two signals worth the industry's attention. First, speculative decoding is evolving from "guessing one at a time" to "intelligent batch guessing + dynamic verification," with the engineering maturity of lossless acceleration continuing to improve; from Medusa to EAGLE to DSpec, each generation of solutions is seeking a better Pareto frontier among acceptance rate, system overhead, and deployment complexity. Second, DeepSeek open-sourcing the complete toolchain and enabling anyone to reproduce it is equivalent to making "verifiable performance" a component of technical credibility—this is more convincing than any one-sided marketing figures, and it echoes the broader backdrop of academia's "reproducibility crisis": when a technical claim can be independently reproduced by anyone on any hardware, its credibility is upgraded from a "claim" to a "fact." For developers hoping to efficiently deploy models like Qwen on local or low-cost GPUs, DSpec offers a practical and viable path to inference acceleration.

Key Takeaways