DeepSeek Boosts Inference Speed by 85%: How Speculative Decoding Is Reshaping the AI Speed Race

A Revolution in Inference — All About Placing Bets

Imagine there's a casino inside DeepSeek's servers. Every time you ask its AI a question, the "dealer" places up to five bets on the next word you're about to see. And recently, DeepSeek published the exact mathematical formula that lets the "dealer" always win — boosting inference speed for the world's cheapest frontier AI by up to 85%.

Here's the key point: the model didn't change, the GPUs didn't change, the answers are mathematically equivalent, and even the probability distributions are identical. Nothing got smarter — they just learned "when to bet."

This article is based on the paper jointly published by DeepSeek and Peking University, with founder Liang Wenfeng listed among the authors: DS Spark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. This isn't a leak or a rumor — the PDF is sitting right there in their open-source repository.

Why Are Large Models So "Slow"

A behemoth like DeepSeek V4 Pro has 1.6 trillion parameters, yet it can only spit out one token at a time.

What Are Tokens and Forward Passes?

A "token" is the basic unit large language models use to process text, typically corresponding to a word, a word root, or a punctuation mark. Take the BPE (Byte-Pair Encoding) tokenizer commonly used by the GPT series: "reasoning" might be split into two tokens, while "AI" is usually a single token. A "forward pass" is the complete computation a neural network performs from input to output: data flows sequentially through the embedding layer and dozens or even hundreds of Transformer attention blocks, ultimately producing a probability distribution for the next token. For a model with hundreds of billions of parameters, a single forward pass requires trillions of floating-point operations. The autoregressive generation pattern means every token requires triggering a full forward pass independently — this is the fundamental source of large-model inference latency, and the core constraint speculative decoding tries to sidestep.

Generating one word fragment requires running a full forward pass, so 10,000 tokens means 10,000 round trips. Absurdly, during each round trip, those massive GPUs spend most of their time "waiting" — and there's a deep hardware reason behind this.

Memory Bandwidth: Where the Real Bottleneck Lies

The fundamental reason large-model inference is slow is the "memory bandwidth bottleneck." Modern GPUs have powerful floating-point compute capabilities, but during the autoregressive inference phase, generating each token requires loading all of the model's weights from video memory (HBM) into the compute cores, and the speed of reading those weights is limited by memory bandwidth rather than raw compute. This state is called a "memory-bound" operation: the GPU's compute units spend most of their time waiting for data transfer, with extremely low utilization. Take DeepSeek V4 Pro's 1.6 trillion parameters — storing the weights alone requires terabytes of memory, and every forward pass reads through all of them. This is precisely the physical basis for speculative decoding's speedup: when batch-verifying multiple tokens, weights only need to be loaded once, amortizing the bandwidth cost and dramatically boosting actual GPU utilization.

It's like owning a super casino but only letting one customer through the door at a time.

The entire inference speed problem of modern AI can be distilled into a single line of math:

Latency = (Drafting Time + Verification Time) ÷ Tokens Accepted Per Round

Every acceleration trick you've heard of is essentially pulling on one of these three levers.

Speculative Decoding: The Small Gambler and the Big Dealer

To solve the inference speed problem, the industry invented an elegant trick — speculative decoding. Instead of having the large model write word by word, hire a small, fast "gambler" (the draft model) to place bets ahead of time: "I bet the next five tokens are…" Then the large model — the "dealer" — verifies all five bets simultaneously in a single forward pass. One round trip is worth five.

Every correct bet pays out; at the first incorrect bet, the dealer sweeps away that chip and all chips after it. The most commonly misunderstood point here is: this is not an approximation.

The Rigorous Guarantee of Mathematical Equivalence

The most counterintuitive property of speculative decoding is its "mathematical losslessness" — it's not an approximation algorithm, but a strictly equivalent acceleration scheme. This guarantee comes from the "rejection sampling correction" mechanism proposed in a 2023 Google DeepMind paper. Specifically, when a token proposed by the draft model is rejected by the large model, the system doesn't simply skip it — instead, it resamples according to the difference between the large model's and draft model's probability distributions, precisely compensating for the distributional bias introduced by the draft model. The final output token sequence is statistically identical to what the original large model would produce through independent autoregressive sampling. This property is crucial: it means speculative decoding can seamlessly replace existing inference stacks without re-evaluating model safety or alignment properties, with extremely low deployment risk — one of the key reasons the industry adopted it so quickly.

The acceptance rule mathematically preserves the large model's exact output distribution. The same checkpoint, the same answer — the only thing that changes is speed.

Two "Broken" Gamblers

The problem is that there are two kinds of gamblers on the market, and both have flaws:

• The Cautious Gambler (Autoregressive Drafting): like Eagle 3. It places one bet, takes a look, then places the next. Each bet builds on the previous one, so quality is high — but betting time grows with the number of chips, forcing it to stay small and shallow.
• The Reckless Gambler (Parallel Drafting): like DeepSeek's own D-Flash. It throws down all chips at once, incredibly fast because drafting is cheap, and it can even afford a deeper "brain." But each of its bets can't see the others.

The fatal weakness of the second gambler is what the paper calls "multimodal collision." Suppose a sentence could have two different endings — two individually valid answers get blended into gibberish. This causes the quality of later bets to decay rapidly: in chat scenarios, the acceptance rate slides from 72% at the start of a block all the way down to 63% at the end. The dealer eats this loss.

DS Spark's Three Aces

The First Ace: A Hybrid Gambler and the "Markov Head"

DeepSeek's first move was to build a hybrid gambler: keep the reckless gambler's deep, parallel brain (one forward pass, all chips at once), but add a tiny sequential "whisper" — which the paper calls the Markov head. As chips fall from left to right, each bet gets a gentle nudge from the previous one, and the collision disappears.

The Engineering Implications of Low-Rank Decomposition and the Markov Assumption

DS Spark's Markov head uses a "Rank 256 low-rank matrix" to implement sequential conditional dependency — a design borrowed from the core ideas of parameter-efficient fine-tuning fields like LoRA (Low-Rank Adaptation). The intuition behind low-rank decomposition is that a complex mapping in high-dimensional space can often be approximated by the product of two small matrices, drastically reducing parameter count while preserving key information. The term "Markov" comes from mathematician Andrey Markov; the Markov assumption states that the current state depends only on the previous state, not the full history — which is exactly how the Markov head works: the k-th draft token only needs to "look at" the (k-1)-th, rather than rerunning the full attention mechanism. This first-order dependency is enough to eliminate "multimodal collision" while keeping the extra compute overhead within about 1% — a precise engineering trade-off that solves the core problem at minimal cost.

And this "whisper" is almost free: it's a Rank 256 low-rank lookup table, a "pocket notebook" rather than a "second brain." The data is striking — the hybrid approach starts with a 93% acceptance rate on math tasks and stays high throughout. A two-layer DS Spark drafter beats a five-layer D-Flash, with the "whisper" adding only 0.2%–1.3% latency in exchange for up to 30% longer accepted sequences. In DeepSeek's offline tests, DS Spark is about 30% faster than Eagle 3 and about 18% faster than its own older parallel drafter.

The Second Ace: The Confidence Head as an "Odds Maker"

What separates a gambler from a casino is "knowing the odds." DS Spark introduces an "odds maker" — the confidence head. As each chip hits the table, it's annotated with a probability score: the probability that this bet passes the dealer's verification, given that all prior bets survived. 0.93 stays, 0.81 stays, 0.22 gets swept immediately — no wasting the dealer's time.

But neural networks have long suffered from "overconfidence."

Why Do Neural Networks Need Calibration?

The "overconfidence" problem in neural networks is long-standing: the softmax probabilities a model outputs often overestimate its own accuracy — a prediction claiming 95% confidence may have an actual accuracy of only 70%. "Temperature scaling" is the simplest and most effective post-processing calibration method: before the softmax, divide the logits by a scalar temperature T. When T > 1, the probability distribution flattens (lowering confidence); when T < 1, it sharpens. The key is that this operation doesn't change the argmax result — it doesn't affect the model's prediction ranking — yet it substantially improves the reliability of the probabilities. DS Spark's "sequential temperature scaling" is a variant that learns an independent temperature value for each position in the draft sequence, because later tokens are naturally more uncertain and require more aggressive compression. Only calibrated confidence scores can become the trustworthy "odds" the scheduler relies on, dramatically improving the entire system's decision quality.

To this end, DeepSeek uses sequential temperature scaling to calibrate the odds maker position by position, until the annotated odds match reality — reducing the calibration error from as high as 8% to about 1%. On chat, the messiest workload, pruning low-confidence chips raised the acceptance rate from 45.7% to 95.7%.

The Third Ace: The "Floor Manager" Almost No One Reported On

This is the most brilliant — and most overlooked — part of the paper. The odds at a single table mean nothing; you have to look at the entire casino floor.

Batch Inference: Why the "Floor Crowd" Matters

"Batching" is a core technique for improving throughput in GPU inference systems. The GPU's parallel architecture is naturally suited to processing multiple requests simultaneously: stitch several users' inputs into a single batch, and a single forward pass can generate multiple answers at once, significantly boosting hardware utilization. However, speculative decoding introduces new complexity in batch scenarios: different user requests have different draft acceptance rates, and if you generate the same number of draft tokens for all requests uniformly, high-confidence requests get "slowed down" by low-confidence ones. DS Spark's hardware-aware scheduler is designed precisely for this problem — it dynamically allocates the draft budget at a global level, essentially solving a real-time resource allocation optimization problem. This kind of co-optimization between "continuous batching" and speculative decoding is a core frontier of current LLM inference engineering, and the main battleground for frameworks like vLLM and TensorRT-LLM to keep evolving.

A real serving system isn't one user but hundreds of requests simultaneously sharing the same batch of GPUs. Verifying an extra chip is only free when the floor is empty; when the floor is packed, every extra chip the dealer checks steals compute from another paying customer.

So DS Spark introduces a "floor manager" — the hardware-aware prefix scheduler. At startup, it profiles real hardware performance into a cost table, then every round it pulls all chips from all tables across the floor, globally ranks them by survival odds, and admits them one by one until the expected return across the entire floor stops rising. On an empty floor, each customer gets four, five, six chips through; on a packed floor, only the safest bets are kept, and the budget automatically tightens. The old system, MTP-1, was hard-coded to a fixed two chips, but now the "dealer" dynamically adjusts with the crowd.

One detail that reflects engineering rigor: at production speeds, the scheduler can't even wait for the current step's confidence score (that would stall the GPU pipeline), so it schedules using predictions from two steps prior. And it's precisely this delay that guarantees mathematical losslessness — the admission decision never peeks at the token it's admitting. This is a "causal firewall."

Real Gains and the Boundaries of Open Source

How Much Did the Dealer Actually Win

Here is DeepSeek's production data comparing against its own older baseline on real user traffic:

• V4 Flash (284 billion parameters, the workhorse): 60%–85% faster single-user generation speed at the same total throughput
• V4 Pro (1.6 trillion parameters, the flagship): 57%–78% faster
• Under medium latency targets, total throughput per GPU improves by about 51%

As for the 661% figure making the rounds, the honest version is this: it only appears at one extreme speed tier where the old baseline had basically collapsed and could barely sustain a single batch. It's not a universal multiplier — what it really means is that "a speed tier that simply didn't exist before now exists" — the frontier has been pushed forward. It's also worth adding that no third party outside DeepSeek has yet independently reproduced these serving figures, and this cautious principle should apply to any lab's announcement.

A New Open-Source Paradigm: Give Away the Algorithm, Keep the Casino

DeepSeek open-sourced a repository called DeepSpec under the MIT license, gathering thousands of stars in its first week. It's a complete draft-model training and evaluation stack, including DS Spark, the older D-Flash, and Eagle 3 as a baseline, bundled with training checkpoints for Open Qwen and Gemma. NVIDIA released official training recipes within days, and you can even plug it directly into vLLM via a speculative config flag.

But read the fine print carefully: the V4 production-grade drafter and custom serving kernels that power that 60%–85% speedup are not in the box. You can reproduce the offline science, but you can't clone their casino.

The Business Logic of "Strategic Open Source"

DeepSeek's "give away the algorithm, keep the casino" open-source model is a textbook example of the "strategic open source" strategy widely adopted by tech companies in AI in recent years. Its business logic is this: releasing papers and foundational code quickly builds technical credibility, attracts top researchers, and accelerates ecosystem prosperity (like official NVIDIA support and vLLM integration) — the spillover effects far exceed the value of the code itself in terms of brand equity. What truly forms the moat is the customized production-grade inference kernels, the operator libraries deeply tuned for their own hardware clusters, and months of accumulated online traffic scheduling experience. These "hidden assets" cannot be reproduced from open-source code. Similar patterns have historical precedent: Google open-sourced the MapReduce paper, but the actual engineering implementation of its GFS cluster was never disclosed; Meta released the LLaMA weights, but its internal inference infrastructure remains a black box. Open source has become a win-win tool for technology leaders to establish industry standards while maintaining competitive barriers.

This is the new unwritten rule of open source — DeepSeek gives away the algorithm and keeps the floor.

The AI Speed Race: The Battlefield Is Quietly Shifting

Step back, and this is the real story. The rules of the AI race are changing: what was once a competition over "whose model is smartest" is now a competition over "who can deliver intelligence to the most people fastest, per dollar of silicon."

DeepSeek V4 Pro's output pricing is about $1.70 per million tokens — roughly 1/17 of GPT-5.5's output rate — and it has even introduced peak-and-off-peak pricing like a utility company. Inference speed is the underlying logic behind all of this — DS Spark lets each GPU serve more users, so idle compute can be converted into discounts. And for AI Agents that burn tokens all day, inference speed is directly your profit margin.

So here's the perspective to leave you with: the next time any lab claims "our AI got faster," just ask one question — did the brain change, or did the casino get smarter? From now on, you'll always be able to tell the difference.

Key Takeaways