DeepSeek Open-Sources DeepInfer Inference Framework: A Single Cluster Handles 1.55 Million eTokens Per Day

DeepSeek open-sources production inference framework DeepInfer: high throughput, cross-hardware, free for commercial use.
DeepSeek has open-sourced DeepInfer, the inference framework actually used in its online services. It already supports large-scale inference for DeepSeek V4.1 Flash, processing 1.55 million eTokens per day per cluster while balancing high throughput and low latency. It supports both NVIDIA and Huawei Ascend without tying users to a single compute platform, and uses the Apache 2.0 license, permitting free commercial use and modification. The article notes that inference framework efficiency directly determines AI service compute costs and power consumption — previously a core engineering capability closely guarded by each company. This open-sourcing effectively hands the key cost-reduction tool to the industry.
DeepSeek recently did something that has drawn considerable attention in AI engineering circles: it open-sourced the inference framework powering its own production services, naming it DeepInfer. This is not an experimental project — it is the very same foundation behind DeepSeek's real online services, already supporting large-scale inference for DeepSeek V4.1 Flash, with a focus on high throughput and low latency.

How Much Work Can DeepInfer Actually Handle
According to a related Bilibili video, DeepInfer's cluster processing capacity is quite substantial — a single cluster can process 1.55 million eTokens per day. This scale shows it is not a lab toy but an industrial-grade framework validated under real business pressure. Achieving high throughput and low latency simultaneously is itself the hardest part of inference systems engineering.
For ordinary users, these numbers may just seem "very large"; but for those building AI services, it means the same hardware can serve more requests, and the cost structure of a single inference run is fundamentally different.
eToken is an engineering measure of LLM inference throughput, referring to the number of tokens the model actually processes during inference (typically including both input and output), with the "e" prefix emphasizing that it is the equivalent token volume actually consumed. Unlike training, which counts samples or sequence length, token throughput in inference scenarios directly corresponds to GPU compute and memory bandwidth overhead, making it the most intuitive indicator of inference framework efficiency. Processing 1.55 million eTokens per day per cluster means the scale of model forward computation the cluster completes daily; converted to per-card or per-request dimensions, this allows further estimation of unit compute cost and peak concurrency — a key basis for evaluating whether a framework can support production traffic.
Not Tied to Hardware: Supports Both NVIDIA and Huawei Ascend
Another characteristic of DeepInfer is its openness in hardware adaptation. Source material shows it does not only support any single chip — it supports both the NVIDIA and Huawei Ascend platforms. This is critical in real deployments: if an inference framework is deeply tied to one hardware vendor, users' options are locked in and migration costs become very high.
By covering both mainstream technology paths, DeepInfer hands more of the decision of "which chip to run on" back to users. For teams that need to deploy services across different compute environments, this unbound design itself reduces decision risk.

Huawei Ascend is Huawei's self-developed AI compute platform, comprising the Ascend series NPU chips and the accompanying CANN software stack (analogous to NVIDIA's CUDA). Since the CUDA ecosystem has long dominated, supporting Ascend in an inference framework typically requires adaptation at the operator layer, memory management, and communication libraries — considerable work. DeepInfer covering both CUDA and CANN paths indicates a hardware-decoupled design at the abstraction layer, allowing the same inference logic to be compiled or scheduled to different backends. For domestic teams, this means a deployment path that does not depend on a single vendor when compute supply is constrained or mixed deployment is needed.
Apache 2.0 License: Free Commercial Use, Modify as You Like
What developers care about even more than technical metrics is the license. DeepInfer uses the Apache 2.0 license, which means you can use it commercially for free, modify it yourself, and use it freely. For enterprise users, this clears legal concerns; for teams wanting to build on it for secondary development, Apache 2.0 also provides ample room.
The license choice of an inference framework often determines whether it can truly enter production environments. DeepInfer went straight for the most open tier.
Apache 2.0 is the most commercially friendly tier among OSI-approved open-source licenses: it allows users to freely use, modify, and distribute the software, including in closed-source commercial products, and does not require derivative works to be open-sourced under the same license (unlike GPL-style copyleft licenses). It also includes explicit patent grant clauses, so users will not be passively infringing due to contributors' patent claims — important for enterprise legal assessment. The only major obligation is to retain copyright and license notices and to mark changes in modified files. Because of these terms, Apache 2.0 is often regarded as the "passport" for infrastructure projects entering production environments.
Why an Inference Framework Is Worth Getting Excited About
Models determine how smart AI is, but the inference framework determines whether you can afford to use it. With the same chips, a slightly less efficient framework means a dramatically higher cost.

The efficiency of the inference framework directly determines the compute cost and power consumption of AI services. This is why technology at this level has traditionally been closely guarded by AI companies — it is the core engineering secret to running models cheaply and fast. Model weights can be open-sourced, but how to make models run efficiently under real traffic has always been each company's hidden trump card.

The Significance of Open-Sourcing: Handing Over the Key to Lower Costs
By making DeepInfer public, DeepSeekAd has effectively handed over the key to lowering costs. For the entire industry, this means more teams can drive down inference costs without reinventing the wheel, bringing intelligence to more people more cheaply.
Perhaps this is the whole significance of open-sourcing DeepInfer: not just another framework, but turning "being able to afford to run it" from a secret of a few companies into public infrastructure everyone can use.
One-line Summary
DeepSeek open-sources its production inference framework DeepInfer: high throughput, cross-hardware, free for commercial use.
Paragraph Summary
DeepSeek has open-sourced DeepInfer, the inference framework actually used in its online services. The framework already supports large-scale inference for DeepSeek V4.1 Flash, processing 1.55 million eTokens per day per cluster while balancing high throughput and low latency. On hardware, it supports both NVIDIA and Huawei Ascend without tying users to a single compute platform; it uses the Apache 2.0 license, permitting free commercial use and modification. The article notes that inference framework efficiency directly determines the compute cost and power consumption of AI services — previously a core engineering capability closely guarded by each company. This open-sourcing effectively hands the key cost-reduction tool to the industry, letting more teams lower inference costs without reinventing the wheel.
Related articles

Daybreak Red Deployment Guide: A Reference Architecture for Supervised Security Investigation
A breakdown of the Daybreak Red reference architecture: security control design, deployment configuration process, and runtime monitoring for supervised security investigation.

Gemini 4 Neon Tops the Models Engineering Index
Gemini 4 Neon tops the Models Engineering Index, sparking Hacker News debate over ranking credibility and evaluation methods. Here's what it means for developers.

Apple's Self-Developed AI Agent: An Agent That Can Operate Your Mac Is Coming
Apple is reportedly developing an AI agent that can operate a Mac, targeting Meta Muse, Claude, OpenAI Dots, and others. This article analyzes Apple's system-level advantages, the competitive landscape, and likely rollout paths in the agent race.