DeepSeek Open-Sources DeepInfer: Inference Cost Reduction and the Shifting Landscape of Domestic Chips

DeepSeek has open-sourced its internal inference framework, DeepInfer. This isn't an ordinary engineering open-source move—it directly ties structural choices made during training to the cost structure at inference time, turning "low cost" from an operational trick into an intrinsic property of the model architecture.

What Was Open-Sourced: A Restrained Scope with Clear Boundaries

The official rationale is straightforward: training lets a model learn, but inference is what puts the model to use, and inference efficiency directly determines compute cost and power consumption. This open-source release includes two things: the DeepInfer inference framework (compatible with NVIDIA and Ascend) and the DeepSeek^Ad Recipe (encoding/decoding components, aligned with the official API).

But the scope is quite restrained: it covers only one model (DeepSeek V4.1 Flash), two hardware types (NVIDIA SM100 and Ascend 950), and only offline inference. KV Cache storage, PD disaggregation, load balancing, and online serving are not included. It's more like a single engine within a production system than a service framework you can deploy out of the box.

This restraint is precisely what's worth reading into. DeepInfer was originally based on vLLM, but traces of vLLM are now minimal—the scheduler, input preparation, and sampling input construction have all been rewritten in C++. In other words, this isn't handing the community a "just make it run" demo; it's extracting the core engine validated in production so you can see where the boundaries lie.

Low Cost Is Structural, Not the Result of Stacking Hardware

The Q&A section provides some production numbers: a single cluster handles about 155 trillion tokens per day, translating to roughly $38 in revenue per GPU per hour. Decode uses EP64 to EP128, Pre-fill uses only EP2 to EP8, KV Cache is stored on 3FS with 10+2 erasure coding, retained for 3 to 7 days.

Behind these numbers is a key judgment: Pre-fill is compute-bound, Decode is memory-bandwidth-bound, and deploying them separately means neither drags the other down. The most cost-saving parts are all tied to model structure—cross-layer KV sharing, where the latter half of layers borrows the KV from the middle layers (i.e., YOCO, from a Microsoft 2024 paper). The first half generates a single global KV, and the latter half reuses it directly; Pre-fill can also terminate early. This must be designed in during training for the savings to materialize at inference.

The biggest takeaway from reading the code is: DeepSeek's low cost is structural. KV is compressed so small, and Pre-fill computation is cut by nearly half, relying on hardware-software co-design (Codesign)—something that can't be achieved by merely stacking hardware or tuning kernels. Global KV cache, PD disaggregation, cross-tier cluster load balancing, and fault handling were not released this time, but it's still a major boon for cloud providers—there's a reference and a ready-made implementation from day one. The result is that for tasks where open-source models suffice, the floor on serving prices will continue to drop.

YOCO (You Only Cache Once) is a decoder architecture proposed by Microsoft Research in 2024. In a standard Transformer, each layer must maintain its own KV Cache, and memory usage grows linearly with the number of layers; YOCO instead lets the first half of the model's layers compute normally and generate a single global KV cache, which the latter half of layers directly reuse without storing their own. This dramatically compresses total KV Cache size and significantly reduces memory pressure in long-context scenarios. More importantly, since the latter layers no longer need to recompute KV, the Pre-fill phase can terminate early, cutting computation by nearly half. But this benefit cannot be obtained on the fly at inference time—the model must be designed with the YOCO structure during training, otherwise there's no cache to reuse at inference. This is the technical foundation for the article's judgment that "structure determines cost."

Domestic Chips: Ascend Becomes a First-Class Citizen, the Bar Rises for Other Vendors

There are only two paths for hardware abstraction: NVIDIA's CUDA and NCCL, and Ascend's CANN and HCCL. In DeepInfer, Ascend is a first-class citizen just like NVIDIA—there are 30 TileLang operators on each side in one-to-one correspondence, plus Ascend versions of DeepGEMM and DeepEP.

Previously, domestic chip vendors maintained their own forks of vLLM or SGLang and were always playing catch-up. Now, with the same engine and the same algorithms, the only gap left is hardware and low-level operators. There are two chain effects: first, the TileLang Ascend backend has been validated at massive production scale, making the CANN ecosystem more convincing; second, supporting only the 950 also helps Huawei push customers to upgrade. DeepSeek plus Ascend is likely to become one of the default combinations for large-scale inference deployment in China.

Other vendors won't have it so easy. For Cambricon, Hygon, Kunlunxin, Moore Threads, MetaX, Biren, Enflame, and Iluvatar CoreX to achieve full compatibility, they would roughly need to overhaul their existing engines and fill in quite a lot: 30 operators for the TileLang backend contract, FP8×FP4 GEMM fusion, Mega MoE that merges dispatch and computation, sparse attention and TopK, plus an expert-parallel library for device-initiated communication, Ngram cross-card lookup tables, and platform-layer adaptation. The more mature the TileLang backend becomes, the lower the porting cost.

TileLang is a domain-specific language (DSL) for tensor computation, aiming to let operator developers write one piece of code and compile native implementations for different hardware backends. It sits between low-level programming models like CUDA/HIP and high-level frameworks like PyTorch, using a compiler to lower a unified tensor program into the target hardware's instruction sequence. For DeepInfer, TileLang is a key layer of hardware abstraction: NVIDIA and Ascend each maintain 30 one-to-one corresponding operators, so the same algorithm logic can run on both chips. The maturity of the TileLang backend directly determines the porting cost for new hardware platforms—the more complete the backend, the fewer operators chip vendors need to add, and the shorter the adaptation cycle. This also explains why the main battleground for adaptation is shifting from hand-written kernels to DSLs and compilers.

Three New Expectations for Hardware

This engine also encodes expectations for hardware. First, low-precision formats: expert weights use FP4, activations and KV use FP8, and scaling factors use UE8M0. Back in V3.1 last year, DeepSeek already said that UE8M0 FP8 was designed for next-generation domestic chips, and now it's fully implemented—chips that don't support it will lose either precision or performance. E8M0 has only an 8-bit exponent and no mantissa, so scaling factors are all integer powers of 2, making hardware implementation very cheap.

Second, memory pressure is reduced: KV per token is as small as under 1KB, dramatically lowering HBM demand for long contexts, which directly benefits domestic chips with smaller HBM.

Third, the bottleneck shifts: expert parallelism at EP64 and above plus Mega MoE require large-scale high-speed interconnect and device-initiated communication, so the bottleneck moves to interconnect, and host bandwidth becomes more important. The Ngram table is placed in host memory, pulled remotely by hash during forward passes, and the NPU path specifically calls Ascend interfaces to lock pages.

UE8M0 is a minimalist scaling-factor number format: all 8 bits are used for the exponent, with no mantissa bits and no sign bit. This means it can only represent integer powers of 2 (e.g., 2^-3, 2^5) and cannot represent arbitrary decimals. In FP8/FP4 mixed-precision computation, scaling factors are used to normalize values of different magnitudes into the representable range of the low-precision format; representing scaling factors with UE8M0 means hardware only needs to perform shift operations, without complex multipliers, making implementation extremely cheap. The trade-off is that the scaling granularity must be fine enough to approximate the true dynamic range using powers of 2. DeepSeek proposed during the V3.1 period that UE8M0 FP8 was designed for next-generation domestic chips, and now fully implementing it in DeepInfer means chips that don't support this format will directly lose out on precision or performance.

Mega MoE refers to fusing the three originally separate steps of "dispatch—compute—combine" in an MoE (mixture-of-experts) model into one large fused operator for execution. Standard MoE inference requires first dispatching tokens according to routing results to the devices hosting each expert, having each expert compute, and then combining the results back—involving multiple kernel launches and cross-device communication. Mega MoE merges these steps, reducing launch overhead and memory round-trips for intermediate results, while better overlapping communication with computation. Under large-scale expert parallelism at EP64 and above, this fusion places higher demands on interconnect bandwidth and device-initiated communication, and is the direct basis for the article's judgment that "the bottleneck shifts from compute to interconnect."

Implications: The Main Battleground for Adaptation Is Shifting

For large-model companies, inference cost increasingly depends on structural choices made during training: KV size, Pre-fill computation, and speculative decoding acceptance rate are all design targets. For chip vendors, joint optimization is now mandatory—Day-0 support directly affects customer choices. The main battleground for adaptation is shifting toward DSLs like TileLang and low-precision formats.

For ordinary developers, DeepInfer won't replace vLLM and SGLang—it doesn't do prefix caching or online serving—but it's a good reference: which states must be stored, which can be recomputed, and how general-purpose frameworks will absorb these designs are all worth following closely.