DeepSeek Open-Sources DeepInfer Inference Framework: Serving V4.1-Flash

DeepSeek has released the inference framework it runs in production for DeepSeek-V4.1-Flash. The project, called DeepInfer, is open-sourced under the Apache 2.0 license, targets two hardware platforms—NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs—and currently offers only offline inference capabilities. The news comes from a MarkTechPost report.

What Exactly Is DeepInfer

DeepInfer's positioning is clear: it is not a general-purpose inference toolkit for developers, but the very inference stack DeepSeek^Ad uses for its own online services. The official pitch centers on "serving DeepSeek-V4.1-Flash production traffic," which means the framework embodies scheduling, memory management, and kernel implementations that have been validated under real workloads—not prototype code from a lab.

The Apache 2.0 license deserves a separate mention. It permits commercial use, modification, and redistribution, making it relatively friendly to enterprise users and reducing the legal friction of secondary development. For an inference framework, a permissive license is often the first step toward building an ecosystem.

Hardware Strategy: Betting on Both NVIDIA and Ascend

The most striking aspect of DeepInfer is that it doesn't focus solely on NVIDIA for hardware adaptation. The framework explicitly supports NVIDIA SM100 GPUs—the compute capability level corresponding to the Blackwell architecture—while also listing Huawei Ascend 950 NPUs as a target platform.

This "dual-stack" strategy is uncommon among open-source inference frameworks. Most projects either bind deeply to the CUDA ecosystem or only adapt to a single domestic chip vendor. DeepSeek's choice to walk on two legs suggests, on one hand, that its internal inference workloads need to cover different hardware environments, and on the other, that Ascend 950 holds real weight in its deployment roadmap. For teams looking to deploy large-model inference in domestic compute environments, this adaptation information alone is valuable for reference.

That said, supporting two heterogeneous hardware stacks means double the maintenance cost. Whether the community can keep both hardware lines updated in sync after taking over is a key indicator of the project's longevity.

SM100 is the compute capability identifier NVIDIA uses in the Blackwell architecture, corresponding to data-center-grade GPU products (such as the B100 and B200 series). Compared with the previous-generation Hopper SM90, SM100 brings significant improvements in FP8/FP4 low-precision compute, memory bandwidth, and NVLink interconnect bandwidth, and introduces hardware features more friendly to larger-scale expert parallelism. For inference frameworks, adapting to SM100 means keeping up with new instruction sets, tensor core layouts, and memory hierarchies rather than directly reusing kernels optimized for A100/H100. This is why DeepInfer emphasizes SM100 support: it signals that the framework has rewritten key kernels for the latest hardware generation.

Ascend 950 is a next-generation product in Huawei's Ascend AI processor roadmap, targeting large-model training and inference scenarios. Unlike the CUDA ecosystem, Ascend uses its self-developed Da Vinci architecture and CANN software stack, with operator development, graph compilation, and memory management differing substantially from the NVIDIA ecosystem. Porting an inference framework to Ascend typically requires rewriting low-level kernels, adapting HCCL collective communication, and handling differences in graph mode and operator fusion. Therefore, "supporting Ascend 950" is not a configuration switch but requires independent engineering investment, and its actual maturity must be verified through measured throughput, operator coverage, and precision consistency.

Offline Inference Only: A Deliberate Trade-off

DeepInfer currently offers only offline inference and does not support online serving scenarios. At first glance, this limitation seems like a shortcoming, but it actually reveals the framework's design focus.

Offline inference is typically used for batch generation, data synthesis, evaluation benchmarking, and similar tasks—insensitive to latency but demanding high throughput and memory efficiency. By open-sourcing the production-validated offline path first, DeepSeek hands over the most mature and most easily reproducible part while avoiding the scheduling logic in online serving that is too deeply coupled with specific business needs.

For users, this means DeepInfer is better suited in the short term as a foundation for batch inference tasks rather than a direct replacement for online inference engines like vLLM or SGLang. If the goal is to build real-time conversational services, its capability boundaries still need to be evaluated.

The core difference between online and offline inference lies in scheduling objectives. Online serving operates on a per-request basis, pursuing first-token latency and tail latency, and requires continuous batching, preemptive scheduling, and fine-grained KV Cache paging to handle the uncertainty of request arrival. Offline inference operates on a per-task basis, where input scale is known and large batches of requests can be orchestrated at once to maximize GPU utilization and throughput. The two differ in design orientations for batching strategy, memory allocation, and fault recovery, so a framework polished for offline scenarios typically cannot directly meet the latency requirements of online serving.

The Competitive Landscape of Open-Source Inference Frameworks

The inference framework space has become quite crowded over the past two years. vLLM established an ecosystem advantage with PagedAttention, SGLang differentiated itself in structured generation and scheduling, and TensorRT-LLM binds to NVIDIA for ultimate performance. What does DeepInfer rely on to gain a foothold at this point?

Its differentiation is likely not in generality but in "deep synergy with DeepSeek models." The official description calls it the production inference framework for V4.1-Flash, indicating that the framework contains optimizations specifically tailored to that model's structure. This co-design approach—model and framework developed together—resembles Meta's path of driving inference optimization around the Llama ecosystem: first serve your own models, then spill value outward.

Another variable is Ascend adaptation. Domestic inference frameworks generally lag behind the CUDA ecosystem in maturity on domestic NPUs. If DeepInfer can deliver usable performance on Ascend 950, its appeal in specific markets will be markedly higher than that of general-purpose frameworks.

PagedAttention is a KV Cache management mechanism proposed by vLLM, borrowing from the virtual memory paging concept in operating systems. It divides the KV Cache into fixed-size blocks allocated on demand, eliminating memory waste caused by pre-allocating maximum length and supporting block-level sharing and reclamation. This mechanism significantly improves memory utilization and throughput under concurrent requests and has become a foundational design borrowed by many subsequent inference frameworks. Understanding PagedAttention helps assess DeepInfer's trade-offs in memory management: if its offline path uses different batching and caching strategies, performance comparisons cannot rely on a single metric alone.

Questions Worth Watching

Open-sourcing the framework is only the starting point; several practical questions determine how far it can go.

Completeness of documentation and examples. Production frameworks are often tightly coupled with internal infrastructure. Whether the open-source version can run independently outside DeepSeek's internal environment is the first hurdle the community will encounter.

The actual level of Ascend 950 support. The report only mentions it as a target platform without providing performance data or feature coverage. NPU adaptation depth varies greatly and requires community testing to verify.

Whether the roadmap includes online inference. Offline and online inference have considerably different requirements in scheduling, batching, and KV Cache management. Whether the official team plans to add online capabilities will directly affect the framework's scope of applicability.

Overall, DeepInfer's open-sourcing has clear value for teams focused on inference performance optimization and domestic compute adaptation. It provides a reference implementation from a real production environment, particularly in the dual hardware stack dimension, filling a gap in publicly available information. But treating it as a plug-and-play general-purpose inference solution is premature for now.

One-line Summary

DeepSeek open-sources its production inference framework DeepInfer, supporting a dual hardware stack but limited to offline inference.

Paragraph Summary

DeepSeek has open-sourced DeepInfer, its production inference framework for DeepSeek-V4.1-Flash, under the Apache 2.0 license. The framework supports both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs—a dual hardware stack strategy rarely seen among open-source inference frameworks. Currently, DeepInfer offers only offline inference, positioning it for throughput-sensitive tasks like batch generation and data synthesis rather than real-time conversational services. Its differentiated competitiveness may come from deep co-optimization with DeepSeek models and the potential market value of domestic NPU adaptation. The article also notes that documentation completeness, Ascend support depth, and the online inference roadmap are key variables determining the project's future vitality.