DeepSeek Open-Sources Inference Engine DeepInfer: The Systems Engineering Behind 155 Trillion Tokens

DeepSeek has open-sourced its inference engine DeepInfer on GitHub. This isn't a new model — it's the inference system DeepSeek actually runs in production, and every answer you receive from DeepSeek is computed by it, one token at a time. According to official figures, a single cluster processes 155 trillion tokens per day. The code is derived from vLLM and released under the Apache 2.0 license, but only if you happen to have Blackwell- or Ascend 950-class hardware on hand.

From Parts to Whole Machine: An 18-Month Open-Source Road

This isn't DeepSeek^Ad's first foray into inference infrastructure. In February 2025, they released a string of low-level libraries within a single week — Flash MLA, DeepEP, DeepGEMM, the 3FS storage system — but the full engine was never made available. An April 2025 note explained why: the engine had been forked from vLLM more than a year earlier, had long since been modified beyond recognition, and was deeply coupled with internal cluster management tooling. Even if it had been forcibly extracted, outsiders wouldn't have been able to run it. The decision at the time was to contribute optimizations back to vLLM and SGLang piece by piece.

Eighteen months later, the whole engine has been released after all. Many files still begin with a comment: derived from vLLM, modified by DeepSeek. Currently it supports only one model — DeepSeek V4.1 Flash.

Flash MLA, DeepEP, and DeepGEMM are three low-level components DeepSeek released progressively in early 2025, each addressing a different part of the inference pipeline. Flash MLA is an efficient implementation of Multi-head Latent Attention (MLA), with memory-access optimizations tailored to DeepSeek models' distinctive attention structure; DeepEP is a communication library for Expert Parallelism scenarios, handling the dispatch and return of tokens across GPUs in mixture-of-experts models; DeepGEMM is a low-level operator library for matrix multiplication. All three are "parts"-level components that must be embedded in a complete inference framework to be useful, unlike DeepInfer, which is positioned as a "whole machine."

Two Bottlenecks: Prefill Is Compute-Bound, Decode Is Bandwidth-Bound

A large model is essentially just a giant file full of numbers. To make it talk, you have to repeatedly compute with those numbers: derive the next character from all the preceding ones, append it, then compute the next. For one person this is simple. The trouble is that thousands of people are asking at the same moment — who gets computed first, who gets batched with whom, which GPU the data goes on, where intermediate results are stored. These logistical chores are the inference engine's job.

Every request passes through two stages. First comes Prefill: your question is read in one pass, thousands of tokens computed simultaneously, and GPU compute is fully saturated. Then comes Decode: the answer is written out one token at a time, and each token requires re-reading the weights from memory. Compute sits idle while bandwidth becomes the bottleneck. Most of the optimizations in DeepInfer target these two bottlenecks.

Mixture-of-Experts: Less Compute, Therefore Cheaper

V4.1 Flash is a mixture-of-experts model with 552 billion parameters. Picture a building full of experts: for each incoming token, the router at the door wakes only a few of them while the rest keep sleeping. So only a small fraction of parameters is actually used per token — roughly 8 billion when reading input, about 16 billion when generating. Less compute, therefore cheaper.

But those 500-plus billion parameters have to live somewhere, and a single GPU can't hold them. The solution is to distribute experts across many GPUs, each responsible for a subset — this is expert parallelism. Yet Prefill and Decode call for opposite layouts: during Prefill, each expert naturally receives a large batch of tokens, so there's plenty of work and groups of two to eight GPUs suffice — smaller groups mean less inter-GPU communication; during Decode, every step requires each GPU to read all the experts it holds, so the more experts on a GPU, the slower each step — hence Decode is spread much wider, in groups of 64 to 128 GPUs. Spreading from 8 GPUs to 64 leaves each GPU holding only one-eighth of its original experts.

Hot experts get overwhelmed; a set of load balancers replicates them onto other GPUs and periodically redistributes according to actual traffic. Wider isn't always better, either — beyond 32 GPUs, communication and synchronization overhead begins to cancel out the gains. The official minimum for efficient deployment is 64 GPUs for Prefill and 64 for Decode — 128 GPUs to start.

Expert Parallelism is the core approach for distributed inference of mixture-of-experts models. Since total parameter count far exceeds single-GPU memory, different experts must be allocated to different GPUs. After a token's expert is selected by the router, its hidden state must be sent to the GPU holding that expert for computation, then the result sent back. This process introduces extra inter-GPU communication overhead, and the communication volume scales with how widely experts are distributed. Expert parallelism width therefore requires balancing memory capacity, per-GPU compute efficiency, and communication cost — the fundamental reason Prefill and Decode use different parallelism degrees in this article.

Speculative Decoding with DSpark: Guess Five, Verify as Many as Capacity Allows

Decode produces only one token at a time, and most of each step's time goes to moving data. Speculative decoding works by having a small draft model guess five tokens in a row, then having the large model verify them all at once — collecting however many leading guesses were correct. DeepInfer uses DSpark, whose draft model has only three layers and can guess all five positions in a single pass.

But verification also costs compute, and later guesses are less reliable, so always verifying all five is often wasted work. DSpark therefore assigns each guess a score — how likely it is to be accepted — and the scheduler checks current load: when GPUs are idle, verify more; when busy, verify only the most confident. Official figures show the number of verified tokens per step dropping from five to 3.2–4.4, with the average accepted length barely changing. In throughput terms, roughly 60 concurrent requests per GPU yields a 4%–7% speedup; the higher the concurrency, the more worthwhile, while at very low concurrency it's a slight loss.

Caching and Scheduling: The Less Glamorous Engineering Work

Every token the model reads leaves behind an intermediate result called a KV cache, which can be reused directly the next time the same prefix appears. DeepSeek stores these caches on 3FS, roughly 100 PB of solid-state drives, retaining chat data for 7 days and Agent data for 3 days — just enough to span a weekend.

But V4.1 Flash has another small cache used by sliding-window attention, called the SWA cache, which doesn't go into 3FS and lives only in the memory of the machine that computed it. When a request moves to another machine, the large cache may hit while the small cache doesn't, and a faithful rebuild would push the last 2,560 tokens through 20 decoder layers. DeepInfer chooses to approximate: recompute only the last 128 tokens, and let attention see only back to the recomputation start point. The resulting cache is approximate, but the recomputation cost is fixed — the same whether the context is 100K or 1M. DeepSeek says total Prefill compute is thus nearly halved.

There's another remarkably plain trick: after each GPU step, the CPU must wrap up and schedule the next batch, during which the GPU can only idle. DeepInfer lets the CPU run one step ahead — while the GPU computes step N, the CPU is already scheduling step N+1. This approach comes from SGLang and NanoFlow and is enabled by default. When JSON output is required, format rules are first compiled into a state machine that only permits valid tokens at each step.

The KV cache (Key-Value Cache) is a key mechanism in autoregressive generation. During Transformer decoding, attention computation for each new token requires the Key and Value vectors of all preceding tokens; without caching, the entire prefix would be recomputed every step, with cost growing quadratically with context length. The KV cache retains these intermediate results in GPU memory or external storage so that each step only computes attention for the new token. A cache hit can skip redundant Prefill computation, but cache capacity is limited and requires eviction policies — the 7-day and 3-day retention periods set by conversation type in this article are one such policy. The SWA cache is a lightweight cache specific to sliding-window attention, retaining attention state only for the most recent tokens.

What Was Open-Sourced, and What Wasn't

Can you build your own DeepSeek with it? Not yet. What's released is the offline inference version; the KV cache system, Prefill/Decode disaggregated deployment, and cross-replica load balancing that production serving requires are not included, on the grounds that they're too tightly bound to internal infrastructure. Add to that support for only one model and only the newest GPUs, and forget about running it on a personal computer. Its real audience is cloud providers and engineers working on collaborative inference. By official figures, the system generates roughly $38 in revenue per GPU per hour at current stock prices.

DeepSeek's model weights have long been public; this time, they've handed over how to run them cheaply as well. For the inference infrastructure field, this may be more instructive than another model release.

One-line Summary

DeepSeek open-sources its production inference engine DeepInfer, revealing the engineering optimizations addressing the Prefill and Decode bottlenecks.