DeepSeek Open-Sources Inference Engine DeepInfer: All the Cost-Saving Tricks Laid Bare

Title

DeepSeek Open-Sources Inference Engine DeepInfer: All the Cost-Saving Tricks Laid Bare

Meta Description

DeepSeek^Ad open-sources its inference engine DeepInfer and codec component Recipe. This deep dive breaks down PD disaggregation, three-tier KV cache, SWA bounded replay, speculative decoding, and zero-overhead batch scheduling—plus what the $38 unit economics really means and where open source stops.

Article Content

This time DeepSeek didn't release a new model or climb any leaderboards. Instead, it open-sourced DeepInfer, the inference framework powering its online services, along with a codec component called Recipe. If model weights are a recipe, the inference framework is the kitchen—same recipe, but how many orders a kitchen can handle per hour and how long guests wait can differ enormously. Open-sourcing an engine is far rarer than open-sourcing weights, because the engine holds the real engineering know-how.

Why an Inference Framework Is More Worth Open-Sourcing Than Weights

Many people think open-sourcing a model means downloading weights, but weights are just a pile of numbers—they don't answer questions. In between sits an entire engineering stack: taking orders, queuing, feeding requests into the chip, retrieving results. Tech blogger Baoyu uses the recipe-and-kitchen analogy—recipes are available to everyone, but a kitchen's scheduling, heat control, and workflow are the real competitive edge.

In April 2025, DeepSeek explained why it wouldn't open-source its internal engine: the internal version was heavily customized, hard to port to external environments, and the team had no bandwidth to maintain a large project. By October, that position had been reversed. Alongside DeepInfer and Recipe, the same release included Search and Fetch—search and scraping services for models. Search and Fetch find the ingredients, DeepInfer is the kitchen, and Recipe is the multilingual menu; the three form a single chain.

Behind Search and Fetch lies a cost that nobody usually accounts for: search engines are built for humans. A results page has ads, recommendations, navigation bars, and pop-ups, which a person can skip at a glance—but a model can't. Every extra word occupies context, is a token, and costs money. These two APIs return raw JSON; Search additionally provides relevance scores and source authority, and Fetch returns cleaned Markdown for a URL. Pricing is 5 yuan per 1,000 successful requests, free until midnight on November 1, 2026, and not tied to any model.

DeepInfer's Core Design: PD Disaggregation and Three-Tier Caching

DeepInfer uses the Apache 2.0 license and supports NVIDIA SM100 and Huawei Ascend 950, with an environment of Linux + Python 3.12 + GCC 13 or above. The currently supported model is DeepSeek V4.1 Flash.

A request first passes through Recipe to become tokens, then goes through two phases: in the prefill phase, the model reads the system prompt, conversation history, files, and question in one go, and each token produces a set of intermediate results at every layer—this is the KV cache; in the decode phase, the answer is emitted token by token, and each emitted token uses all the preceding KV cache. The two phases consume completely different resources—prefill is compute-bound, while decode spends most of its time reading parameters and cache from GPU memory.

So DeepSeek splits them onto different machines—this is what's commonly called PD disaggregation. The system was designed with three goals from the start: compatibility with chips from different vendors, organizing prefill and decode around common hardware, and keeping KV cache for at least 72 hours. In deployment, prefill uses common 8-GPU servers, decode uses 64- to 128-GPU supernodes, and the vision encoder is pulled out separately, with first-token latency and inter-token latency optimized separately. The official docs include a rather interesting line: the vision encoder generally needs only a single GPU, and it's recommended to use the cheapest GPU available.

The most cost-saving part is caching. In agent scenarios, every time a tool is called, the client must resend the full history—system prompt, tool descriptions, every previous reply and tool result, none can be omitted. And the vast majority of this was already computed in the previous round. DeepSeek's approach is to store the computed KV cache and only compute the newly added portion next round, provided the prefix is exactly identical and aligned from the very first token.

How dramatic is the effect? Running on the same GPU for a day, without cache hits it's about 60,000 tokens per second; with cache hits it's 1.5 million—a 25x difference. On DeepSeek's online service, the average input per request is 77K, average first-token latency is 400 ms, P99 is 1,000 ms, and cache-hit input is priced at 1/50 of cache-miss input.

This cache isn't stored in the compute nodes' own memory but in a separate storage cluster using 3FS, the distributed file system DeepSeek open-sourced earlier, with a capacity of about 100 PB. KV cache is written in blocks of 128 tokens; agent traffic is retained for 3 days and chat traffic for 7 days—chat traffic has long-tail hit characteristics, with a small amount of cache being read repeatedly, so dropping it hurts overall hit rate more; keeping it only 3 days would reduce chat hit rate by 0.5% to 1%. In industry terms, Anthropic, OpenAI, and Google retain cache for roughly 5 minutes to 24 hours, so 3 and 7 days are quite long.

KV cache is a technique in Transformer inference for reusing intermediate computation results. During autoregressive generation, each time a new token is generated, the model needs to perform attention computation over the Key and Value vectors of all previous tokens. If computed from scratch each time, the computation grows quadratically with context length. KV cache stores these already-computed Key/Value pairs and reuses them directly in the next step, computing only the new token's portion. The cost is that memory usage grows linearly with context—the longer the context and the higher the concurrency, the more memory the cache consumes. This is why cache hit rate has such a huge impact on inference cost: a hit means skipping the massive matrix operations of the prefill phase and going straight to decode.

SWA Bounded Replay: Turning Recomputation Cost into a Constant

Cache always misses sometimes, and this part best reveals the changes in V4.1. Some layers in V4.1 use sliding window attention (SWA), where each token only looks at the 128 tokens nearest to it. The running Block KV cache and State KV cache are stored in the inference GPU's memory, with the persistent portion written to 3FS; SWA separately maintains the first 20 layers, each with a ring buffer window of 128 tokens, plus 6 extra for speculative rollback. This part doesn't go into 3FS—only one copy is kept in the memory of the machine that produced it, cleared after about 3 minutes, with a hit rate above 98%; cross-machine hits pull data over a 400G VLAN.

For the 2% that miss, V4's approach was exact recomputation, restoring window state layer by layer, with the number of tokens to recompute equal to the number of layers times the window length. To reduce this, V4 also wrote this cache to disk—and ended up consuming nearly half of the disk cache's capacity. The problem was that it's only useful for the few minutes a conversation lasts, so using disks that can hold data for days to store it was clearly a poor trade.

V4.1 switched approaches, called SWA bounded replay: only the last 128 tokens are recomputed, and during computation attention is also restricted to within these 128 tokens, yielding an approximate window cache. Recomputation cost becomes a constant, independent of how long the context is or how many layers the model has. The same approach is applied to the Decoder's 20 layers; after the change, instead of processing the last 2,560 tokens, only 128 need processing, cutting total prefill computation by nearly half.

Sliding Window Attention (SWA) is a sparse attention variant: each token attends only to a fixed number of nearest tokens (e.g., 128) rather than the entire history. This changes attention computation and cache overhead from growing linearly with context length to a bounded constant, significantly reducing long-context inference cost. The cost is that the model cannot directly see information outside the window, which usually requires alternating with global attention layers to compensate. SWA cache maintenance is also more complex: the window is ring-shaped, old tokens are evicted as new ones enter, and cache state must be tracked precisely—once lost, the window contents must be recomputed.

Speculative Decoding and Zero-Overhead Batch Scheduling

Caching solves fast reading; decoding solves fast emission. The idea of speculative decoding is to let a lightweight draft module guess 5 tokens in a row, then have the large model verify them all at once, directly accepting the correct guesses. But verification itself costs compute; verifying all 5 drafts wastes a lot of compute on rejected guesses. DeepSeek's in-house solution first estimates the acceptance probability of each guess, tilting the verification budget toward more promising ones; dynamic verification reduces the number of checks per step by about 30%, keeps average acceptance length roughly unchanged, and improves online throughput by 6%. As the sequence count goes from 16 to 96, the gain climbs from -1.5% all the way to 11%—this number is conditional; not all workloads get faster.

Another cost hides between CPU and GPU. After the GPU finishes the current step, it must stop and wait for the CPU to update the previous round's state; for requests generating thousands of tokens continuously, this waiting happens repeatedly. DeepInfer's zero-overhead batch scheduling hides CPU work inside GPU compute time: while the GPU processes step N's forward pass and sampling, the CPU is already preparing step N+1's batch and processing the sampling results from an earlier round. With speculative decoding enabled, a request may accept multiple candidates in a row, and the CPU doesn't yet know how many will ultimately pass—organizing computation with the old length would throw off query length, key length, and rotary position encoding positions entirely. DeepInfer maintains an in-flight count, reserves cache space in advance when allocating resources, and then has device-side kernels correct the lengths.

V4.1's deployment scale is prefill EP2 to EP8 and decode EP64 to EP128; with such wide expert parallelism, GPUs in the same communication domain must stay in lockstep. DeepInfer adds in-group voting to the scheduler, with two modes: one requires all GPUs to meet the condition before switching, the other allows partial GPUs to trigger, preventing small requests from waiting too long. After voting there's a second round of confirmation, because having requests in the queue doesn't mean this round can actually be scheduled.

Speculative Decoding is a technique for accelerating autoregressive generation. It first uses a lightweight "draft" model (or module) to quickly generate multiple candidate tokens, then lets the target large model verify these candidates in parallel all at once: if the draft guessed the prefix correctly, multiple tokens are accepted at once, reducing the number of large-model forward passes. The key is that verification is parallel, costing far less than generating one by one. The higher the draft quality and the longer the acceptance length, the more obvious the speedup; conversely, if many candidates are rejected, verification compute is wasted. So how to allocate verification budget and dynamically decide how many candidates to verify is the core of engineering optimization.

Doing the Math: $38 Is Revenue, Not Profit

DeepSeek unusually disclosed its unit economics: for a cluster processing 155 trillion tokens a day, converted at peak pricing, it earns an average of $38 per GPU per hour; times 24 and times 30, that's about $27,000 per GPU per month. Liang Wenfeng once said at a closed-door investor meeting that API pricing is reasonable profit—roughly recouping the cost of a batch of equipment in 10 months—earning only reasonable profit, not pursuing profit maximization.

Others are doing different math: one blogger, extrapolating from the maximum squeezing capacity of GB200 NVL72, calculated that a quarter's worth of compute could generate $100 billion to $149 billion in token revenue per year, nearly 10 times higher than institutional forecasts; another, using a B300 rental price of $7 per hour with sufficiently high utilization, calculated an 82% gross margin. These numbers sound great, but they all need to be discounted—the first two rest on optimistic assumptions, and the third deducts only GPU rental, ignoring storage, networking, operations, and R&D costs, and depending heavily on cache hit rate and traffic structure. Treating them as DeepSeek's profit margin or payback period doesn't hold; $38 refers to revenue.

Unit Economics refers to the analytical method of calculating revenue and cost per single measurable unit (e.g., per GPU, per million tokens, per request). In AI inference services, "revenue per GPU per hour" is commonly used to measure hardware monetization efficiency, but it reflects only top-line flow, not profit. To get true profit, one must also deduct GPU depreciation or rental, storage and network bandwidth, power and cooling, operations staffing, R&D amortization, and waste from idle rates. Thus the same set of numbers can yield wildly different gross margins under different assumptions—which is why outside interpretations of DeepSeek's cost data diverge so dramatically.

What's Open-Sourced Is the Engine, Not the Chassis

The official ReadMe states clearly: building a complete inference service also requires an independent KV cache system, decoupling facilities between the vision encoder and decoder, and load balancing across data-parallel replicas. The deployment FAQ also acknowledges that production cache management and load balancing are too deeply coupled with internal infrastructure and have not been fully open-sourced for now. External teams getting the code can see how requests pass through the scheduler, cache, communication, and sampling, and can extract a particular optimization to bring back to their own systems, but replicating an equally fast and cost-efficient online service still requires integrating storage, networking, service management, and failover themselves.

Hardware is also a hard threshold: the current support list includes only NVIDIA SM100 and Huawei Ascend 950; H100, H800, A100, and consumer GPUs are not included. The official recommendation is 64 GPUs for prefill plus 64 for decode—this is the starting scale for achieving efficiency, not the minimum to get it running. The Ascend part deserves separate mention: at the end of September, DeepSeek had already published programming tools, compute libraries, and communication libraries for the Ascend platform, with the Huawei team participating in development; Bloomberg reported that DeepSeek plans to deploy at least 160,000 Ascend 950DT chips in a new data center in Inner Mongolia, but this is a reported plan and cannot be counted as deployed scale. Recipe's performance tests were conducted only on NVIDIA SM100; having both chips on the support list does not mean they deliver the same performance and cost.

What's Really Worth Watching Here

Over the past few years, the outside world has seen only scattered parts: Flash MLA for attention, DeepGEMM for matrix operations, DeepEP for expert communication, EPLB for expert placement. Each part has papers and benchmarks, but how a request passes through them and what the data contracts in between look like has remained a black box. DeepInfer puts these into a single public project; although the full production control plane isn't provided, the assembly diagram is now visible.

This also shows that model architecture is determining runtime architecture. MLA absorption, sliding windows, consumer-producer layering, hyper-connection MOE, multi-token prediction—these aren't configuration items a general-purpose engine can automatically infer; they directly change cache layout, scheduler state machines, computation graph shapes, and rollback logic. That's why DeepSeek had to write its own engine rather than patch a general-purpose framework.

Open-sourcing itself is neither as naive as idealism nor as dark as conspiracy theories suggest. Recipe turns official processing logic into a public library; third parties implement it accordingly, and behavior naturally aligns—good for developers, who face fewer issues when switching providers; even better for DeepSeek, as prompt formats, tool-call formats, and streaming output rules become the de facto standard as more third-party servers adopt them. Search and Fetch compete for the entry point—every time an agent looks something up, it must call a search tool, and whoever is the default occupies the position. What the public recipe brings in return is worth far more than the recipe itself.

When inference engines and interface formats are made into public goods, what's left that can truly differentiate model vendors? Is it data, scenarios, or the ability to put a freezer in every store like Coca-Cola?

One-line Summary

DeepSeek open-sources inference engine DeepInfer and Recipe, revealing the engineering ledger behind PD disaggregation, KV cache, and scheduling optimizations.

Paragraph Summary

DeepSeek has open-sourced its inference framework DeepInfer and codec component Recipe, along with Search/Fetch search and scraping services, forming a complete chain from entry point to kitchen. The article explains the PD disaggregation architecture that splits prefill and decode onto different hardware, the 100PB-scale 3FS-based KV cache that boosts post-hit throughput 25x, SWA bounded replay that turns miss recomputation cost into a constant, in-house speculative decoding that dynamically allocates verification budget for 6% throughput gain, and zero-overhead batch scheduling that hides CPU-GPU waiting. It also clarifies that $38/GPU/hour is revenue, not profit, and notes that the production chassis—cache management, load balancing—is not open-sourced, with hardware limited to SM100 and Ascend 950.