DeepSeek Open-Sources DeepInfer: The Cost-Saving Tricks Are There, the Scheduling Isn't

DeepSeek has open-sourced its production inference engine DeepInfer under the Apache 2.0 license, and unusually also published official operational data: a single cluster processes 155 trillion tokens per day. By the company's own reckoning, that works out to $38 in revenue per GPU-hour, versus a market rental price of roughly $8 for a B200—a gap of more than four times. A quick reality check first: this is revenue versus rental cost, not profit. Electricity, depreciation, and storage haven't been deducted.

But what's really worth examining in this release isn't that four-fold number—it's the dividing line between what was opened and what wasn't.

Open-Sourcing a Framework Is Not Open-Sourcing Scheduling

Many people conflate open weights with open frameworks. Open weights give you a recipe you can follow; an open inference framework gives you the stove, the assembly line, and the pre-processed ingredients. But what DeepInfer opens up is the craft of "how to cook this dish," not the scheduling system for "how to serve 100,000 people at once."

The official explanation: modules like KV cache management and cross-replica load balancing are too deeply coupled with internal infrastructure to be cleanly separated out. So the central dispatch room stays locked. That explains how DeepSeek^Ad drove costs down—but it also means nobody else can reproduce the same numbers.

Three Cost-Saving Tricks: The Full Craft Layer, Handed Over

The three optimizations DeepInfer discloses are the core of this cost structure.

PD Disaggregation: Separate Teams for Prep and Plating

Pre-fill (processing input) consumes compute, while Decode (generating output) consumes memory bandwidth. Put both on the same GPU and they compete for resources. DeepSeek splits the two phases: 8-GPU servers handle Pre-fill, while 64-to-128-GPU supernodes handle Decode, with a GPU ratio of roughly 1:1. The analogy is splitting prep work and plating into two teams, each doing its own job.

The Resource-Profile Difference Between Prefill and Decode

The key to understanding PD disaggregation is that the bottleneck for these two phases lies in entirely different hardware dimensions. The Prefill phase performs attention computation over the entire input sequence, with compute cost growing linearly—or even quadratically—with input length. It is a classic compute-bound task with high Tensor Core utilization. The Decode phase generates only one token per step but must repeatedly read the entire KV cache and model weights, so data movement far exceeds computation. It is a memory-bound task.

If the two are co-located on the same GPU, Prefill's long computations block Decode's iteration cadence, causing noticeable jitter during generation; conversely, Decode's sustained bandwidth usage slows Prefill's batch processing efficiency. Once separated, each type of hardware can be configured with its own optimal batch size and parallelism strategy. That's why DeepSeek assigns 8 GPUs to Prefill and 64-to-128-GPU supernodes to Decode—the latter needs larger-scale high-bandwidth interconnect to carry KV cache transfers across GPUs.

KV Cache: The Hit-Rate Business of SSD Clusters

In chat scenarios, cache is retained for 7 days; in agent scenarios, 3 days. The average request carries 77,000 input tokens, with a 96% cache hit rate. On pricing, a hit costs 2 cents per million tokens versus 1 yuan for a miss—a 50x difference.

Here's an easily overlooked point: a cache hit isn't free. Storage, bandwidth, and garbage collection all cost money—they're just amortized.

KV Cache and Prefix Caching

During LLM inference, generating each token requires attention computation over all previous tokens. To avoid recomputation, the system caches the Key and Value matrices at every attention layer—this is the KV cache. Its size grows linearly with context length and concurrency, and in long-context scenarios it can occupy tens of GB, making it the primary consumer of GPU memory.

A "cache hit" refers to the prefix caching mechanism: when the beginning of a new request exactly matches a historical request, the already-computed KV can be reused directly, skipping that portion of Prefill computation. Agent workloads, multi-turn conversations, and fixed system prompts naturally share large prefixes, so hit rates can be very high. Every point of hit rate improvement saves real Prefill compute—which is fundamentally why DeepSeek can price a miss at 50x a hit: the marginal cost structures of the two are completely different.

Speculative Decoding with Dspark: Accurate Guesses Are Fast Savings

The draft model guesses 5 tokens at a time, and a confidence head determines which are worth verifying. The number of draft tokens verified per step drops from 5 to between 3.2 and 4.4, saving 30%. Combined with sliding-window attention, the model only looks back a short distance: where exact reconstruction would require dragging the last 2,560 tokens through 20 layers, it now recomputes only 128—nearly halving Prefill computation.

Dspark boosts single-user speed by 60% to 85%, with production measurements of 230 to 300 tokens per second—300 characters per second for coding—and 0.4-second time to first token. But when it comes to high-concurrency throughput, the gain is only 4% to 7%. Single-user experience and cluster throughput have never been the same thing.

How Speculative Decoding Works

Standard autoregressive generation produces one token per step and must wait for the previous token to finish before computing the next, severely wasting GPU parallelism. Speculative decoding works by using a smaller, faster draft model to guess multiple candidate tokens at once, then having the large model verify the whole sequence in parallel. Verification is parallel: a single forward pass determines whether each position is accepted, so as long as the guesses are accurate enough, one large-model forward pass can yield multiple tokens.

The key is that verification must guarantee the output distribution matches what the large model would sample on its own—otherwise quality suffers. This is achieved through acceptance-rejection sampling. The more accurate the draft, the higher the acceptance rate and the greater the speedup; an inaccurate draft wastes verification overhead. Dspark's "confidence head" dynamically trims the number of draft tokens verified per step—essentially an adaptive trade-off between acceptance rate and verification cost. This also explains why single-user latency improves markedly while high-concurrency throughput gains are limited: under high concurrency the GPU is already saturated, so the serial steps saved by speculative decoding don't translate directly into throughput.

What 77,000 Input Tokens Reveals

An average input length of 77,000 tokens aligns closely with agent scenarios—agents resend the full context on every tool call. But the company didn't disclose the agent share, and long contexts could also come from codebases and documents, so don't overstate the case.

In July of this year, US enterprises' token processing through Chinese models on OpenRouter briefly reached 58%, but contributed only about 10% to 30% of spending. DeepSeek alone accounted for about 16% in June. This shows that Chinese models selling cheap doesn't equal low cost; a low revenue share is a pricing issue, not proof of a cost advantage. Using this to infer a cost floor is still a logical step short.

What Wasn't Opened Is the Real Moat

Three areas remain closed: KV cache management, full online orchestration for PD disaggregation, and cross-replica load balancing. The open-source version is currently offline inference—anyone wanting to build an online service must supply their own caching and routing.

Translated: the parts that make one machine fast are yours; the parts that make ten thousand machines cheap are kept. The official line is that the coupling is too deep, not a deliberate withholding—but the result is that the most valuable piece, cluster scheduling, remains in DeepSeek's hands.

This isn't unique to DeepSeek; it's the boundary every company open-sourcing a production system faces: craft can be shared, but operational know-how at scale is hard to share. For users, DeepInfer's value lies in providing a production-validated reference architecture for inference optimization. For DeepSeek, open source amplifies ecosystem influence without handing over the real cost moat.

As for whether this is generosity or holding back—the answer depends on which side you're standing on.

Why the Scheduling Layer of Inference Services Is Hard to Open-Source

The scheduling layer of an online inference system must solve resource allocation across multiple replicas, requests, and tenants: which replica a request routes to, which storage tier the KV cache lives on, how to migrate when a replica is overloaded, and how to keep long-tail requests from dragging down overall latency. These decisions depend on real-time monitoring data, cluster topology, hardware heterogeneity, and business priority policies—and are often deeply tied to a company's monitoring systems, deployment pipelines, and capacity-planning tools.

Extracting this into a general-purpose open-source component requires abstracting an interface decoupled from internal infrastructure, with high cost and maintenance burden. This is industry standard: open-source frameworks like vLLM and SGLang provide a scheduling skeleton, but genuine production-grade global routing, elastic scaling, and failover are largely built in-house by each company. So DeepInfer's open craft layer is valuable for reproducing single-machine optimizations, but reaching the officially published cluster-level cost still requires users to fill in the scheduling piece themselves.