Full Analysis of DeepSeek's Open-Source Inference Framework DeepInfer: Inside LLM Inference Optimization

DeepSeek has open-sourced its internal inference framework, DeepSeek-Infer, sparking widespread discussion in the community. With mature solutions like vLLM and SGLang already dominating, why would DeepSeek build and open-source its own inference framework? What unique optimizations does it offer? And what problems do these optimizations solve? Bilibili creator Muxian's video series provides an in-depth analysis. Based on that sharing and official information, this article takes you through it.

Why DeepSeek-Infer?

An inference framework is the bridge between models and hardware. Simply put, it's responsible for getting the model running on hardware: how exactly to compute, how multiple GPUs cooperate, the scheduling order of requests—all of this needs to be arranged by the framework. DeepSeek^Ad-Infer powers DeepSeek's large-scale online services, and its design goals target the core pain points of production environments.

According to the Bilibili creator, DeepSeek-Infer is not reinventing the wheel but rather deeply customized for its own model and hardware characteristics. For example, it specifically optimizes model execution, sampling, and operators, and considers multi-GPU work partitioning and expert load balancing. These optimizations are often difficult to achieve—or not taken far enough—in general-purpose frameworks.

Core Optimization Techniques Explained

Computation and Multi-GPU Coordination

DeepSeek-Infer implements several optimizations at the computation level: model execution, sampling, and specially optimized operators. In multi-GPU scenarios, it focuses on how to partition work and how to distribute expert load more evenly. This is especially important for Mixture-of-Experts (MoE) models, where uneven expert load leads to wasted computational resources.

Speculative Decoding and State Recovery

The framework employs Speculative Decoding: it first generates some draft tokens and then verifies them, confirming more content in one pass and thereby accelerating generation. It also supports sliding-window limited replay: during state recovery, only a small trailing segment is replayed, reducing redundant computation.

Speculative Decoding works on the principle of "draft with a small model, verify with the large model." Since autoregressive generation with a large model requires a full forward pass for every token, the cost is high; speculative decoding instead uses a lighter draft model (or a shallow/small-batch path of the same model) to guess several tokens in a row, then has the target large model verify these drafts in parallel in one pass. During verification, draft tokens are accepted as long as they match the target model's distribution; at the first mismatched token, the draft is truncated and the process falls back to the target model's sampling result. The benefit is compressing multiple serial forward passes into a single parallel forward pass, significantly boosting throughput when the draft hit rate is high; the cost is wasted compute when draft quality is poor. Therefore, the degree of distribution alignment between the draft and target models, and the draft length, are key tuning parameters.

Sliding-window limited replay targets state recovery scenarios in inference: when a request is preempted, batches are reordered, or a rollback is needed, the traditional approach recomputes the entire context from scratch, which is costly. Limited replay only replays tokens within a recent small window, reusing intermediate states cached before the window, thereby reducing redundant computation while guaranteeing correct results. This is closely tied to KV Cache reuse mechanisms and is an important means of reducing tail latency in long-context scenarios.

One-step Scheduling

One-step Scheduling means that while the CPU arranges the next step of tasks, the GPU can continue computing, minimizing waiting. This helps reduce idle time between CPU and GPU and improves overall throughput.

One-step Scheduling addresses the CPU-GPU pipeline of inference systems: the scheduler on the CPU decides which requests to run in the next batch and how to form batches, while the GPU executes forward computation. With a serial "compute then schedule, schedule then compute" pattern, the GPU idles during CPU scheduling. One-step scheduling overlaps scheduling decisions with GPU computation—the CPU prepares tasks for the next step in advance while the GPU continuously executes the current step, hiding scheduling overhead. In large-scale online services, although single-step scheduling takes little time, it accumulates and noticeably affects throughput and tail latency—which is why it is regarded as a key optimization.

Batch Invariance and Determinism

These two properties relate to "cash-out" (likely referring to consistency of paid services). Batch invariance ensures that the same request produces consistent model results regardless of which batch it is computed in; determinism controls the randomness of token selection, so that the same input under the same conditions yields the same output. Both have specific conditions of use, but they safeguard the stability of online services.

Batch Invariance solves the "intra-batch interference" problem. Multiple requests on a GPU are packed into a batch for parallel computation; the tiling and reduction order of matrix multiplication vary with batch composition, causing subtle numerical differences for the same request across different batches, which can in turn change sampling results. Batch invariance fixes the computation graph and reduction order so that a single request's output is independent of its batch, which is crucial for online services that require reproducible and reconcilable results. Determinism further constrains the sampling step: given the same input, same random seed, and same conditions, the selected token is exactly the same. Only with both working together can paid services behave stably during retries, audits, and A/B comparisons.

Structured Output

The framework supports structured output, for example requiring the model to reply in a specified JSON format, which can be achieved through constraints. This is very useful for downstream applications that require strict formats.

Open-Source Scope and Missing Components

What DeepSeek-Infer open-sources this time is the offline inference portion. The official documentation lists three parts not included in the repository: the complete KVCache system, vision encoding and Prefill/Decode disaggregated deployment, and request load balancing across multiple replicas.

However, most of the engine capabilities needed for disaggregated deployment are already implemented in the repository. The Bilibili creator suggests analyzing the code together with officially published deployment information, distinguishing what has been opened up versus what still needs improvement, before discussing its advantages.

Prefill/Decode disaggregated deployment is the mainstream architectural evolution direction for LLM inference today. The Prefill phase processes the entire prompt at once—compute-intensive with high parallelism; the Decode phase generates token by token—memory-access-intensive and latency-sensitive. The two have different preferences for hardware resources, and co-locating them on the same instance causes mutual interference. Disaggregated deployment places the two types of workloads on different node pools, scaling and scheduling them separately, which improves overall GPU utilization and reduces first-token latency and generation jitter. The KVCache system is the core of the Decode phase: it caches the Key/Value tensors of historical tokens to avoid recomputing attention at every step. Its capacity management, prefix reuse, cross-request sharing, and eviction policies directly determine the cost of long contexts and high concurrency.

Supported Models and Hardware

Currently, DeepSeek-Infer supports the DeepSeek V4.1 Flash model, with hardware support for NVIDIA SM100 GPUs and Huawei Ascend 950. SM100 corresponds to GPUs with compute capability 10.0 in NVIDIA's Blackwell architecture, such as B200 and GB200; it is an architecture identifier rather than a specific GPU model.

Recipe and Token Adder: The Request Processing Pipeline

DeepSeek Recipe is also open-sourced alongside. A paragraph a user sends to the API cannot be used for computation as-is; Recipe organizes it into the input the model needs, and together with Token Adder converts it into Token IDs; after generation, it organizes the model's output into the API response. The computation in between is handled by the inference backend.

Directions Worth Exploring

The Bilibili creator lists several leads for further analysis: Why must Prefill and Decode be deployed separately? What exactly does KVCache cache, and how does it eliminate redundant computation? What optimizations has DeepSeek made at the KVCache layer? How do Recipe and Token Adder turn requests into model inputs? Why might speculative decoding actually be faster? These principles will help you understand where DeepSeek-Infer optimizes, why it does so, and under what circumstances it has advantages.

Conclusion

The open-sourcing of DeepSeek-Infer gives the community a window into a major player's inference optimization practices. It makes targeted designs in computation, scheduling, consistency, and more. Although the complete system is not fully open yet, the open-sourced offline inference portion is enough for developers to learn its core ideas. As subsequent videos are released, we will gain a deeper understanding of the technical principles and applicable scenarios behind these optimizations.

One-line Summary

DeepSeek open-sources its internal inference framework DeepSeek-Infer; an analysis of its custom optimizations and open-source boundaries.

Paragraph Summary

DeepSeek has open-sourced DeepSeek-Infer, the internal inference framework powering its large-scale online services, drawing community attention. Based on Bilibili creator Muxian's video series and official information, this article explains why DeepSeek built its own framework despite mature solutions like vLLM and SGLang: it is deeply customized for its own models and hardware, covering model execution, sampling, operator optimization, multi-GPU partitioning, and expert load balancing. Core techniques include speculative decoding to accelerate generation, sliding-window limited replay to reduce redundant computation, one-step scheduling to hide CPU scheduling overhead, batch invariance and determinism to ensure online consistency, and structured output constraints. The open-source scope covers only offline inference; the complete KVCache system, vision encoding, Prefill/Decode disaggregated deployment, and multi-replica load balancing are not yet included. It currently supports DeepSeek V4.1 Flash, with hardware support for NVIDIA SM100 and Huawei Ascend 950; the accompanying Recipe and Token Adder handle request-to-token conversion.