DeepSeek Open-Sources Inference Framework DeepInfer: The Secret to Halving AI Inference Costs

DeepSeek has once again laid the industry's cards on the table. After open-sourcing model weights, it has now open-sourced DeepInfer, the core inference framework powering its online services, along with the companion DeepSeek Recipe components—and attached a detailed "ledger." Two numbers in that ledger stand out: a single B200 GPU can generate $38 in revenue per hour, while market rental is only about $8; and on average, 96% of the input received online doesn't need to be recomputed at all. This article avoids jargon and follows the path of a single message from pressing Enter to the first character appearing on screen, to see how DeepSeek's "central kitchen" actually drives inference costs down.

Why the whole industry erupted over an open-source inference framework

Many people conflate "open-source model" with "open-source inference framework." What news usually calls open source is mainly model weights—the hundreds of GB of trained parameters, plus model structure, tokenizer, and some inference code, and you can run the model. The more accurate term is "open weights," equivalent to publishing the recipe for a signature dish. But DeepSeek^Ad doesn't make one dish a day; it handles massive requests daily. With the same recipe, a home kitchen makes ten portions a day, while a central kitchen makes a hundred thousand—and each portion costs less. The difference isn't the recipe; it's how the kitchen schedules orders, divides labor, reuses semi-finished components, and keeps burners from idling. That central kitchen is the inference framework.

This matters for three reasons. First, it's one of the most valuable engineering secrets of a large-model company—the same weights deployed by different teams can cost several times more or less. Just last month, OpenAI halved the API price of GPT-6 Luna, citing exactly cache and inference efficiency improvements—today's price war is fought on inference efficiency. Second, DeepSeek previously stated it wouldn't open-source the full engine, because the internal inference engine was too tightly coupled to its own cluster and the team lacked the bandwidth to maintain it; this time it delivered on that promise. Third, it didn't just give code—it attached the ledger: a single cluster processes 155 trillion tokens per day, with average revenue of $38 per GPU per hour. Note this is revenue, not profit, and it's calculated at off-peak discounted prices; costs weren't disclosed, but compared to market rental, the margin is easy to sense. What does 155 trillion mean? About 1.8 billion tokens per second on average. No wonder a commenter said: the real OpenAI has arrived.

Cost-saving tactics seen through the journey of a single message

Stop 1: Tokenization's anti-counterfeiting and speedup

Models can't read text—only numbers. The first thing a message goes through is translation into a string of numbers, i.e., tokens. This step looks simple, but DeepSeek specifically open-sourced DeepSeek Recipe to handle several practical issues. The first is anti-counterfeiting: internally, the model uses special tokens to mark dialogue structure—where the user speaks, where thinking ends, where tools are called. What if someone manually types an identical token string to impersonate a system instruction? Recipe's approach is much like an official seal—real tokens have dedicated IDs; the same characters you write in text are treated as ordinary text, no matter how similar, and aren't the seal. Conversely, tokens spelled out as ordinary text in model output aren't treated as real instructions.

The second is speedup. A single message today can easily be tens of thousands of characters, and translating it one by one from start to finish takes time. Recipe found that the ordinary text segments between special tokens can be split and translated in parallel, then reassembled in order. Another detail: a Chinese character typically occupies three bytes and may be split across several tokens; Recipe first assembles incomplete characters before emitting them, avoiding garbled text on screen. More importantly, this logic aligns exactly with DeepSeek's official API, so when self-deploying DeepSeek models, the interface behavior matches the official one.

Stop 2: Separating reading the question from writing the answer

Have you noticed that AI always pauses before answering, then types out one character at a time? Because large models work in two completely different phases. The first is Pre-fill, which you can think of as reading the question: all the content you send—system prompt, chat history, uploaded files, your question—is read in one pass; that pause you see is it reading. The second is Decode, writing the answer, and it can only write one token at a time, looking back at everything before each token.

These two phases have completely different temperaments. Reading the question processes tens of thousands of tokens simultaneously; it's about compute—the more compute the chip has, the better. Writing the answer produces one token at a time; the chip spends most of its time not computing but fetching data—moving parameters and valuable records from VRAM—so it's about memory bandwidth. Assigning both workloads to the same machines is like making one chef both chop a thousand pounds of vegetables and plate dishes one by one—neither gets done well. So DeepSeek splits them onto different machines, known in the industry as PD separation. Reading uses common 8-GPU servers; writing uses supernodes connecting 64 to 128 GPUs; if images are sent, separate machines understand the image first. After separation, each can be optimized independently: the reading side focuses on shortening your wait for the first character, the writing side focuses on making characters appear fast and steady.

Memory access characteristics of Pre-fill vs. Decode

Pre-fill processes the entire input; tens of thousands of tokens can be computed in parallel, matrix multiplications are large and compute-dense, GPU compute units are nearly saturated, making it a classic compute-bound workload. Decode generates one token per step; although it passes through all model layers, the vectors involved each time are tiny, and most time is spent moving weights and KV Cache from VRAM to compute units, leaving compute units largely idle—a classic memory-bound workload. The two phases have almost opposite hardware demands: the former wants maximum compute, the latter wants maximum memory bandwidth. The engineering significance of PD separation is that each type of machine can be procured and scheduled at its optimal configuration, rather than using the same machines to accommodate both workloads, ensuring neither runs at full utilization.

Stop 3: Teaching the model to "flip through notes"

Why does chatting with AI get slower and more expensive the longer you talk? Every new message you send, the model rereads the entire conversation from the start. After 50 rounds, round 51 rereads the previous 50. With coding AI agents it's worse—every tool call resends the entire history, system prompt, tool descriptions, and previously read code. DeepSeek's online data shows an average input of 77,000 tokens per request; rereading from scratch every time would explode costs.

So there's a key technology called KV Cache—think of it as the notes the model takes while reading: intermediate results computed at each layer for content already read are stored; next time the same prefix comes, it flips through the notes and only reads the new part. The rule is the prefix must be identical to use the notes; repeated content starting mid-stream doesn't count. How far has DeepSeek taken this? On average, 96% of online input directly hits the notes and needs no recomputation. By comparison, in February last year that number was 56%. The same GPU doing reading, counting only what actually needs computing, processes 60,000 tokens per second; including note hits, it's 1.5 million per second—a 25x difference. This is directly reflected in the price list: input that misses cache is $1 per million tokens, input that hits cache is $0.02—a 50x difference. So now you know: AI agents can run cheaply largely thanks to these notes.

Where are all these notes stored? DeepSeek's answer is to build a dedicated archive, using its own open-source distributed file system 3FS to build an independent storage cluster of about 100PB of SSDs—roughly equivalent to 110,000 1TB SSDs. Agent notes are kept for 3 days, ordinary chat for 7 days. Why keep chat longer? Because a few notes in chat get flipped through repeatedly; losing them hurts. Data shows that if chat were kept only 3 days, hit rate would drop 0.5 to 1 percentage point. Notes aren't placed directly in compute machines' memory but in a separate archive for two reasons: first, any machine can look up any note, so it doesn't matter which machine your next message is routed to—scheduling is free; second, compute machines fail an order of magnitude more often than storage—if notes are in the archive, they survive machine failure. And flipping through archives mustn't slow the entrance: the entrance spends only a little time checking the directory to confirm how long a hit will be, while actually moving data from disk to GPU happens in the background concurrently with computation. Note that this archive's code wasn't open-sourced this time; DeepSeek says it's too deeply tied to internal infrastructure, but the design is described in great detail in the technical report.

DeepSeek V4.1 has another clever design. Some layers in its model are "nearsighted," looking only at the nearest 128 tokens; these notes are only useful during the few minutes of a conversation. The previous generation V4 also archived these short-term notes, which took nearly half the space with little benefit. V4.1 changed this: short-term notes aren't archived, only kept in local memory, cleared after about three minutes, and hit rate remains above 98%. If a miss occurs, it doesn't reread from scratch—it only rereads the last 128 tokens to reconstruct an approximate note. Thus the cost of recovery is fixed, whether the conversation is 10,000 or 1,000,000 characters. The same idea applies to another part of the model's layers: originally each reading recomputed the last 2,560 tokens; now only 128, cutting total reading computation roughly in half. This technique is called SWA Bound replay, and it's included in the open-sourced DeepInfer.

The spatial cost of KV Cache and eviction strategy

KV Cache deserves its own 100PB archive because its size grows linearly with context length. Each layer and each attention head must store a Key and Value vector for every processed token; the longer the context and the higher the concurrency, the more it simply won't fit in VRAM. This is also why improved cache hit rates directly show up in pricing: a hit means those tokens' KV needn't be recomputed, saving both compute and time. Cache can't be kept forever; expiration times are needed for eviction. DeepSeek differentiates retention by business type, essentially trading off hit rate against storage cost—in chat, a few notes get repeatedly hit, so keeping them longer pays off; in agent scenarios, distribution is relatively uniform.

Stop 4: Triage desk and off-peak discounts

Why is DeepSeek more expensive on weekday daytime, and half price at night and weekends? Because compute fears two things most: congestion and idling. DeepSeek's model uses MoE (Mixture of Experts) architecture—think of a hospital where not every doctor is a general practitioner, but there are many specialty clinics, and each token only goes to the clinics it needs. When writing answers, DeepSeek spreads these clinics across 64 to 128 GPUs, each doing only a few experts; each GPU fetches less data, so it's faster and can serve more patients—batching improves efficiency.

But there's a fatal problem: the barrel effect. So many GPUs advance synchronously step by step, and each step waits for the slowest GPU; if any clinic has a long queue, all GPUs wait. So DeepSeek built four triage desks: for reading, for writing, for images, and between experts. For example, hot experts get more clinics—more replicas of popular experts per GPU—and rebalancing based on recent load periodically. The result: the slowest GPU is less than 10% slower than average; DeepSeek claims overall online performance reaches about 80% of a perfectly balanced ideal.

Now idling. Traffic is high by day and low at night, and the ratio of machines needed for reading vs. writing keeps changing, so the system auto-scales, adjusting machine counts on both sides, and deliberately keeps extra machines for sudden spikes. Half-price off-peak essentially encourages you to shift demand to when machines are idle. By the way, inter-expert load balancing is included in the open-sourced DeepInfer; the cross-machine triage desks, like the archive, are not open-sourced this time.

MoE expert parallelism and the load-balancing challenge

MoE architecture splits the feed-forward network into many experts, activating only a few per token, controlling per-inference compute while having a huge parameter count. But in distributed deployment, experts are spread across different GPUs, and one forward pass requires cross-GPU token exchange (all-to-all communication); any overloaded expert on any GPU slows the whole batch. This is the barrel effect mentioned. Solutions generally fall into two categories: replicating hot experts so multiple GPUs each hold a copy to share traffic; and dynamic rebalancing, periodically reassigning expert placement based on observed routing distribution. DeepSeek narrows the gap between slowest GPU and average to under 10%, indicating scheduling close to ideal balance—a key engineering step in driving down MoE inference costs.

Stop 5: Speculative decoding and zero-waste scheduling

Why can AI only type one character at a time—can't it write several at once? In the standard approach, a large model writes only one token per computation step. To go faster, you need to write several per step; here's a clever method called speculative decoding: imagine a senior expert with an intern. The intern is fast and drafts the next five tokens in one go; the expert checks word by word—the first correct ones are used directly, the first error is corrected by the expert, the rest discarded, and repeat. The final result is vetted by the expert, so quality isn't compromised. If the GPU isn't busy, checking five takes about the same time as the expert writing one—pure profit.

But online, one GPU serves dozens of requests simultaneously; each extra draft check consumes compute, and drafts get more likely wrong the further out they go—checking likely-wrong tokens is waste. DeepSeek's in-house solution is called DeSpark, which does one extra step: estimate each draft token's acceptance probability, combine with actual GPU throughput, and compute how many tokens per request is most worthwhile to check. The effect: about 30% fewer draft tokens checked per step, with almost no change in accepted tokens, yielding about 4% to 7% more throughput under online workloads. DeepSeek's July paper also gave a number: in the previous V4 online system, compared to the old scheme, at the same total throughput, each user saw 60% to 85% faster token output.

There's also a zero-waste detail. Between every two computation steps, the CPU schedules processing of the previous step's results, during which the GPU would otherwise idle. DeepInfer lets the CPU work one step ahead: while the GPU computes step N, the CPU has already arranged step N+1, so the GPU never stops. Like a chef still cooking this dish while the waiter has already queued the next table's order and prepped ingredients within reach. The final online result is about 230 tokens per second on average, up to 300 in coding scenarios.

A splash of cold water: the weight and barriers of this open source

The boundaries of this open source must be clear. First, what's open-sourced is the offline inference version, the engine core; the note archive, the separate PD deployment, and the cross-machine triage desks mentioned above are not released—you'd have to build them yourself. Second, the barrier is high: currently only supports DeepSeek's own V4.1 Flash model; hardware must be data-center GPUs like NVIDIA B200/GB200, or Huawei Ascend 950; to get good efficiency you'd start with roughly 64 GPUs for reading plus 64 for writing—forget personal computers. So direct beneficiaries are cloud providers, large enterprises, and developers of open-source inference frameworks like vLLM and SGLang, who can directly study and borrow these optimizations—DeepInfer itself was originally modified from vLLM.

But at least the engine part is the real deal. In the repository's deployment Q&A, DeepSeek says the open-source operator library is largely consistent with the internal version, with end-to-end performance differing by at most 5%. This is significant because for the first time it lays out "how to make AI cheaper" from a black box to everyone. From open-source models to open-source operators, and now open-source inference framework and public ledger, DeepSeek is dismantling the entire chain step by step for all to see. One easily overlooked detail: the official version supports both NVIDIA and Huawei Ascend, putting domestic chips and NVIDIA into the same production-grade system. Previously the competition was whose model is smarter; next it's who can sell the same intelligence cheaper.

DeepSeek has open-sourced the core of its kitchen—what moat do you think it has left?

One-line Summary

DeepSeek open-sources inference framework DeepInfer and publishes its ledger, breaking down how inference costs are driven down.

Paragraph Summary

After open-sourcing model weights, DeepSeek has open-sourced DeepInfer, the inference framework powering its online services, and the Tokenization component DeepSeek Recipe, while publishing its inference ledger for the first time: a single cluster processes 155 trillion tokens daily, with $38 revenue per GPU per hour versus about $8 market rental. The article follows a message's processing path to break down cost savings: Tokenization anti-counterfeiting and parallel speedup, PD separation so reading and writing use different machines, KV Cache letting 96% of online input avoid recomputation, a 100PB independent storage cluster for notes, MoE expert load balancing and off-peak discounts, speculative decoding and CPU look-ahead scheduling. It also notes the open-source boundaries: the note archive, PD deployment, and cross-machine triage desks are not open-sourced; hardware starts at 64+64 data-center GPUs; direct beneficiaries are cloud providers and inference framework developers.