DeepSeek's Open-Source Inference Engine: The Secret Behind 155 Trillion Tokens Per Day

DeepSeek recently open-sourced DeepInfra, the inference engine behind its API, along with a detailed technical report explaining how the parts that weren't open-sourced are implemented. According to DeepSeek, a single cluster running this stack can serve 155 trillion tokens per day—a figure that should make anyone watching AI infrastructure reconsider the ceiling of inference efficiency.

From Single-Machine Inference to Millions of Concurrent Users: Two Very Different Hardware Tasks

When you run a model locally, everything is simple: one user, one prompt, one GPU doing all the work. But serving that same model to millions of people at once completely breaks this setup. Behind every request lurk two hardware tasks pulling in opposite directions.

The first is pre-fill, where the model reads the entire prompt in parallel at once, building its working memory of the conversation—this is also where the KV cache comes into play. The second is decode, where the model writes out the response token by token, each token having to wait for the previous one to finish. To quote the video: "pre-fill is one big batch of math done all at once, while decode is a tiny piece of math repeated thousands of times."

If you let each GPU do both tasks at the same time, they interfere with each other—when a large prompt comes back, everyone else's answers get stuck. DeepSeek^Ad's solution is to separate the two onto different machines: pre-fill is compute-intensive, so it runs on standard eight-GPU machines; decode is memory-intensive, since every token requires reading model weights and conversation memory, so it runs on supernodes made up of hundreds of GPUs. In production, roughly one pre-fill GPU corresponds to each decode GPU.

KV cache (Key-Value Cache) is a core mechanism in Transformer inference. During autoregressive generation, each new token the model produces requires attention computation over the Key and Value vectors of all previous tokens. Recomputing them every time would cost that grows quadratically with context length. The KV cache approach stores the already-computed Key/Value vectors for each layer, reusing them directly in subsequent generation and only computing a new copy for the new token. A cache hit therefore means skipping a large amount of redundant computation. But this cache takes up considerable GPU memory—the longer the context and the higher the concurrency, the larger the footprint—which is precisely the fundamental reason the decode stage is classified as "memory-intensive": the bottleneck isn't compute power, but repeatedly reading model weights and this ever-growing conversation memory.

KV Cache Retained for Days and a 96% Cache Hit Rate

Most companies retain prompt caches for only five minutes, but DeepSeek extends this to several days—three days for agent traffic and seven days for chat. Supporting this strategy is a dedicated data cluster equipped with roughly 100 PB of SSD storage.

The results are striking: according to DeepSeek's data, 96% of all input tokens hit the cache, meaning the model only needs to re-read 4 out of every 100 tokens. This is exactly why the API charges so much less for cached input. For developers building agents, keeping the prompt prefix stable is the cheapest optimization available—this is especially critical in scenarios where the same long context is resent every conversation turn.

Clever Tricks in Pre-fill: Bucketed Scheduling and Sliding-Window Replay

When the new prompt portion enters pre-fill, the load balancer sorts requests by length into four buckets, preventing short prompts from being blocked by a giant 250K-token request. Each GPU has a time budget for every step, and the scheduler predicts how many prompts it can read within that budget.

The most interesting technique in pre-fill is "SWA bounded replay." Some layers of the model use sliding window attention, attending only to the most recent 128 tokens. Rather than computing the entire window from scratch, the system replays only the short window at the end of the prompt—according to DeepSeek, this nearly eliminates total pre-fill work.

Sliding Window Attention (SWA) is a sparse attention variant: each token only attends to a fixed-length local window in front of it (128 tokens here) rather than the entire context, reducing attention computation and cache overhead from growing with length to a constant level. The cost is that distant information beyond the window is no longer directly visible, typically passed indirectly layer by layer through stacking. The key to "SWA bounded replay" mentioned in the article is this: since each layer only cares about that short window at the end, recomputation doesn't need to process the entire prompt from the beginning—replaying only the window at the end of the prompt is enough to restore the required state, which explains why it can nearly eliminate total pre-fill work.

MoE Expert Scheduling and Speculative Decoding on the Decode Side

The decode stage uses a Mixture-of-Experts (MoE) architecture, where each token activates only a small group of specialized subnetworks. DeepSeek spreads these experts across 64+ GPUs, with each GPU holding only a few experts, thereby serving very large user batches simultaneously. The problem is that some experts are far more popular than others, and the slowest GPU drags down the whole group. The solution is to reserve a backup slot on each GPU for a copy of a popular expert, and the system moves these copies based on traffic conditions every few thousand steps.

The final acceleration technique is speculative decoding: a small draft head guesses the next five tokens, and the large model verifies all five in a single step. DeepSeek adds a confidence head on top of this, predicting the probability that each guess will be accepted, so the system only spends compute checking guesses likely to survive. According to its data, confidence scheduling yields about a 6% throughput improvement on top of checking five tokens—at this scale, 6% means a lot of GPUs.

Speculative Decoding exploits the characteristic of the decode stage where "tokens are generated serially and the GPU sits largely idle." It has a smaller, faster draft model first guess several candidate tokens in a row, then the large model verifies these candidates in parallel all at once—the verification costs roughly the same as generating a single token. If the guesses are accepted, it's as if multiple tokens were produced in one step; if rejected, it falls back to the first point of disagreement and continues. Correctness is guaranteed by rejection sampling, so the output distribution matches what the large model would generate on its own, making it "lossless acceleration." The confidence head DeepSeek adds is used to predict the survival probability of each guess, concentrating verification compute on candidates more likely to be accepted.

Batch-Invariant Mode: Even Temperature Zero Isn't Necessarily Reproducible

An easily overlooked detail: even with temperature set to 0, the same prompt on a busy server can yield different answers. The reason is that the GPU chooses different summation methods depending on the number of requests in the batch, and tiny rounding differences can flip which token wins.

DeepInfra offers a batch invariant mode that forces the math to come out the same every time. This mode is off by default because computing the same way every time is slower. If you need reproducible output for evaluating or debugging agents, you should enable this setting. The video author's local experiments confirmed this: the same prompt sent alone versus placed in a batch produced an answer that changed at the 14th token, and repeated exactly every run—showing this isn't random sampling, but that batch composition genuinely affects the output.

The Open-Source Boundary and Hardware Requirements

It's important to be clear about DeepInfra's limitations. The open-source version only supports offline inference and can run on your own computer. The KV cache system and load balancer are tied to DeepSeek's internal infrastructure; currently only descriptions have been provided, and the code has not yet been released.

On the hardware side, these kernels are written for NVIDIA data center Blackwell chips (such as B200) and won't work out of the box on devices using different Blackwell chips, such as the DGX Spark. After examining the source code, the video author found that the attention library is built only for data center chips, and the matrix multiplication library throws an "unsupported architecture" error on other architectures. It's advisable to confirm the project's kernel target architecture before renting hardware—recompiling won't solve this problem.

If you want to run DeepSeek v4.1 Flash yourself, the model already runs on standard engines like vLLM—DeepInfra was originally built on top of vLLM as well. This path of "building on open source and giving back to open source" may be more noteworthy than any single technical detail.

Security Detail: Special Tokens Always Remain Plain Text

The DeepSeekRecipe component is responsible for converting requests in OpenAI or Anthropic format into the tokens the model expects, and converting the output back into a response. This seemingly boring conversion layer hides a security issue: the model uses special tokens to mark the end of user messages and the start of its own response. If a user inputs these special markers, a naive server might convert them into real control tokens, and the model would mistakenly believe its turn has begun—this is one of the common prompt injection techniques.

DeepSeek's implementation ensures that anything the user inputs always remains plain text. The video author's tests showed that when a user inputs the start marker of an assistant turn, naive tokenization produces two assistant turns, whereas after processing through DeepSeekRecipe only one remains, and the input markers become five ordinary tokens.

Special tokens are markers reserved in the tokenizer for the model's control structure that don't correspond to ordinary text, such as symbols marking the start/end of user messages and the start of assistant turns. They are given clear role semantics during training, and the model uses them to determine "whose turn it is to speak now." One prompt injection technique is to trick the server into treating a user-input string as these control tokens, thereby faking the appearance that "the assistant's turn has begun" or "the system instruction has ended," bypassing the original conversation boundaries. The defense principle is straightforward: user content must go through the plain-text channel during tokenization, and any string resembling a special marker can only be split into ordinary tokens, never mapped to real control tokens.

The Economics Behind the Scale

According to DeepSeek's data, each decode GPU produces about 14,000 tokens per second for all its users, while each user still sees streaming output at hundreds of tokens per second. On average, each GPU generates about $38 in revenue per hour—and that's at off-peak prices. One caveat: some people online convert this figure into revenue per gigawatt, but that number is based on off-peak prices, assumes GPUs are fully loaded year-round, and depends on the not-yet-open-sourced KV cache system.

The core philosophy of this system can be summed up in one sentence: serving a model at scale is mainly about not doing work twice, and never letting GPUs sit idle.

One-line Summary

DeepSeek open-sources its inference engine DeepInfra, revealing the engineering details and boundaries behind serving 155 trillion tokens per day.

Paragraph Summary

DeepSeek has open-sourced DeepInfra, the inference engine behind its API, and used a technical report to explain the design of the parts not open-sourced. The article walks through its core techniques: separating compute-intensive pre-fill from memory-intensive decode onto different hardware; retaining the KV cache for three to seven days, achieving a 96% input cache hit rate backed by a roughly 100PB SSD cluster; bucketed scheduling and SWA bounded replay on the pre-fill side; and dynamic MoE expert scheduling plus speculative decoding with a confidence head on the decode side. The article also notes that even with temperature=0, output can still vary due to batch composition, so batch invariant mode must be enabled; the open-source version supports only offline inference, and its kernels are locked to data center Blackwell chips.