DeepSeek Open-Sources Inference Engine DeepInfer: The AI Cost War and the Contest Over National Destiny

From Ban to Open Source: A Dramatic 24-Hour Turnaround

On October 19, 2026, the U.S. Department of Homeland Security issued a public directive requiring all civilian federal agencies to purge from their information systems all DeepSeek^Ad products as well as any third-party services built on DeepSeek technology. The order had actually taken effect back in late July; it was only on this day that it was brought into the open.

Yet the day after the ban became public, at 12:09 PM on October 20, a new repository named DeepInfer appeared on GitHub, with its first commit message consisting of just two words: Initial Commit. Licensed under Apache 2.0, this project is not a model but an inference framework—the very engine DeepSeek actually uses when serving large numbers of users online.

An executive order can block products from entering government networks, but it cannot block the flow of code. Sanctions target the circulation of products; open source releases the circulation of knowledge. The former can be terminated with a signature, but once the latter is released, it can never be taken back.

The Model Is the Blueprint; the Inference Framework Is the Production Line

To grasp the significance of this, one must first clarify the relationship between models and inference frameworks. The model is the blueprint, the weights are the recipe, and the inference framework is the production line. The blueprint determines what you can build; the production line determines how much each unit costs to build. With the same model, swapping in a different engine can make unit costs differ several times over. Over the past three years, the open source community has received only blueprints; the actual production line has remained locked away in the data center.

DeepInfer did not emerge out of thin air. Its LLM class file header explicitly states: 'Derived from vLLM, Modified by DeepSeek.' The project was initially built on top of vLLM and also drew on SGLang and TensorRT-LLM. As for operator libraries like FlashMLA, DeepGEMM, and DeepEP, DeepSeek open-sourced each of them individually over the past two years. The community provided the foundation, and DeepSeek returned the entire building.

This is not an isolated act of generosity, but the compound interest of an open source chain. According to the officially published deployment Q&A, a single cluster processes 1.55 million tokens per day, and during off-peak hours each GPU generates roughly $38 in revenue per hour. This is the first time in history that a frontier lab has completely handed over the very tools it relies on for its livelihood.

vLLM is a high-throughput LLM inference and serving engine open-sourced by a team at UC Berkeley in 2023. Its core innovation is PagedAttention—managing the KV cache in pages much like an operating system's virtual memory, greatly reducing GPU memory fragmentation and thereby significantly boosting concurrent throughput. It quickly became the de facto standard for deploying large models in the open source community, and many commercial services have been built on top of it. DeepInfer's 'Derived from vLLM' label means it inherits this mature scheduling and memory management skeleton, then layers on DeepSeek's own operators and scheduling optimizations. Understanding this is important: DeepSeek did not invent an inference engine from scratch, but rather stood on the community's foundation and gave back the improvements honed in production—precisely the concrete manifestation of the 'compound interest of the open source chain' mentioned in the text.

SGLang is another high-performance inference framework, known for efficient prefix caching and structured generation via RadixAttention; TensorRT-LLM is NVIDIA's official inference optimization library, deeply tied to its GPUs and compilation optimizations. DeepInfer's reference to all three shows that it absorbed mainstream technical approaches such as prefix caching and graph compilation, then made trade-offs tailored to its own traffic characteristics.

Performance Leap: Cache Hits and Scheduling Optimization

The core problem DeepInfer solves is the cost structure of large model inference. Inference has two steps: prefill (reading the context) and decode (generating token by token). The expensive part is that many queries share the same opening segment.

DeepInfer's approach is to retain this memory. In DeepSeek's own traffic, 96% of inputs hit the prefix cache. As a result, single-GPU prefill throughput jumped from 60,000 tokens per second to 1.5 million tokens per second—a 25x increase.

It also optimized two time segments. The first is called DeepSpark, developed in collaboration with Peking University—a small model guesses five tokens first, and the main model then decides which are worth verifying. The number of drafts to verify per batch dropped from 5 to 3.2–4.4, boosting production throughput by 4%–7%. The second is SWA bounded replay: rebuilding the cache originally required recomputing the last 2,560 tokens, but now only the last 128 are recomputed. Total prefill computation is nearly halved, and the cost of long contexts decouples from length.

The third problem is how ten thousand GPUs can work together like a single brain. At each decode step, the CPU must schedule the next batch of requests, and in the past the GPU could only sit idle waiting. DeepInfer lets the CPU schedule the next batch while the GPU is running the current one, enabled by default. Expert replicas are rebalanced once every ten thousand steps.

The most imbalanced GPU differs from the average load by only 9%. More critically, the same input produces exactly the same output regardless of which other requests it is batched with—this is called batch invariance, and it is both reproducible and auditable.

Prefix Caching works on the principle that during large model inference, identical preceding text produces identical Key/Value vectors. If the KV cache corresponding to multi-turn conversations or system prompts is retained, subsequent requests only need to compute the new portion, saving redundant prefill. The fact that 96% of DeepSeek's inputs hit the prefix cache is precisely because a large number of requests share system prompts, few-shot examples, or conversation history. The higher the cache hit rate, the lower the compute cost per token. Only by combining KV cache paging and reuse can prefill throughput be pushed from 60,000 tokens per second to 1.5 million. This also explains why DeepSeek lists its KV cache system as a core component it has not open-sourced—the cache hit rate directly determines cost and is a key operational asset.

Speculative Decoding works by having a small, fast model first guess several candidate tokens, which the large model then verifies in parallel, accepting the correct ones. DeepSpark improves on this by reducing the number of drafts to verify per batch, thereby boosting throughput with almost no loss in quality.

What Is Opened Is the Standard; What Is Kept Is the Operations

DeepInfer is not fully open source. A complete inference service requires three things: a KV cache system, disaggregated deployment of prefill and decode, and cross-instance load balancing. None of these three were open-sourced this time.

In its deployment Q&A, DeepSeek states: the cache uses 3FS storage, 10+2 erasure coding, and roughly 100 PiB of solid-state drives; agent traffic is retained for 3 days, and chat records for 7 days; the smallest unit for efficient deployment is 64 GPUs for prefill and 64 GPUs for decode. This is not a gift for individual developers.

Apache 2.0 means any cloud provider can legally use this engine to run a business, possibly even more cheaply than DeepSeek itself. But the half that truly determines cost remains in its own data center. This playbook closely resembles Google's open-sourcing of Android back in the day—the system is yours, but the service framework stays with me. What is opened is the standard; what is kept is the operations. The engine is yours; the power plant stays with me.

Dual Hardware Track: NVIDIA and Ascend as Equal First-Class Citizens

DeepInfer supports two chips: NVIDIA SM100 and Huawei Ascend 950—not as an afterthought for compatibility, but as equal first-class citizens. For the past decade, NVIDIA's true moat has never been transistors, but the CUDA software ecosystem. Now a frontier lab is telling the world, through production-grade code, that this level of efficiency can also be achieved on Ascend.

Looking back, on September 30 DeepSeek ported TileLang to Ascend, completed jointly with the Huawei team. Even earlier, Bloomberg reported that its data center in Inner Mongolia would deploy at least 160,000 Ascend 950DT chips. This path was made public as documentation for the first time.

The logic of U.S. controls is: without advanced chips, you cannot train frontier models, and you cannot serve users at scale. DeepSeek's answer is: train a model of the same tier with one-tenth the compute, drive service costs to the lowest possible level, and then publish the method. When efficiency can be open-sourced, scarcity is no longer a moat. Inference cost is essentially electricity cost, and China's power generation, according to Musk, may be three times that of the U.S. DeepSeek's announcement says 'reduce compute costs, reduce energy consumption,' which translates to: turn the energy advantage into an advantage in intelligent supply.

CUDA is the parallel computing platform and programming model NVIDIA has built since 2006, including compilers, math libraries, debugging and profiling tools, and a vast developer ecosystem. For more than a decade, the vast majority of deep learning frameworks and operators have prioritized CUDA adaptation, creating a migration barrier where 'switching chips means rewriting the底层.' DeepInfer listing Huawei Ascend 950 and NVIDIA SM100 as equal 'first-class citizens' means its operator libraries, scheduling logic, and performance optimizations have been honed to production grade on Ascend as well, not merely made 'compatible enough to run.' This requires rewriting a large number of low-level kernels and leveraging domain-specific languages like TileLang to reduce cross-chip porting costs. For the industry, this proves that non-CUDA routes can support scaled serving of frontier models, thereby weakening the lock-in effect of the CUDA ecosystem.

Global AI Sovereignty: A Third Path

The significance of this solution extends beyond China and the U.S. For Europe, the Middle East, Southeast Asia, and the Global South, there used to be only two ways to run frontier models: buy American cloud, or buy American chips. Now there is a third: take open-source weights, pair them with an open-source engine, install them in your own data center, and use your own electricity.

The premise of export controls is that the other party must use your infrastructure. When models, engines, and chips form a closed loop, this premise is bypassed. AI sovereignty has gone from a slogan to an engineering blueprint that can be followed and built.

In 1856, the Bessemer converter caused steel prices to fall by more than half within 20 years; only when steel became cheap did railways, skyscrapers, and later forms of warfare emerge. What is being made public today is not a piece of code, but a kind of industrial knowledge—how to make intelligence cheap. When the price of intelligence falls to the level of electricity bills, applications will explode, and the capability comparisons between nations will be reshuffled.

A Sober View: A Standards War Not Yet Won

One should not over-mythologize this. DeepInfer currently supports only one model, DeepSeek V4.1 Flash Upper; the GPUs used are not on the supported list; deterministic sampling will directly assert-fail on Ascend; the open-source version of FlashMLA is missing one assembly optimization, making decoding 5%–10% slower; and all performance numbers are company self-tests, with no third-party verification.

The standard has not yet won; it has merely just been put on the table. What truly matters here is not yet another open-source model, but that open source has for the first time moved from capability to cost. The moat is not the model, nor the chip, but the ability to make intelligence cheap. This time, that ability has been partially made public. The next question is not whose model is strongest, but when intelligence becomes as cheap as electricity, who can afford it, and who will use the most of it.

Batch Invariance means that the same request produces exactly the same output regardless of which other requests it is batched with. In conventional inference, batching changes the reduction order of floating-point operations, causing subtle output differences that make reproduction and reconciliation difficult. DeepInfer claims to achieve batch invariance, meaning the service provider can reliably reproduce any given output, facilitating auditing, billing, and troubleshooting. This is an engineering property highly valued in enterprise-grade deployments, and it is also where its scheduling design differs from ordinary open-source frameworks. However, this feature has so far only been demonstrated in company self-tests and awaits independent third-party verification.

One-line Summary

The day after the U.S. ban became public, DeepSeek open-sourced its production-grade inference engine DeepInfer, pushing competition from model capability to cost.

Paragraph Summary

The article recounts how, after the U.S. issued a ban on DeepSeek in government systems in October 2026, DeepSeek open-sourced its production inference engine DeepInfer (Apache 2.0) on GitHub the very next day. Derived from vLLM and incorporating ideas from SGLang and TensorRT-LLM, DeepInfer uses prefix caching, DeepSpark draft verification, SWA bounded replay, and CPU-GPU pipelined scheduling to boost single-GPU prefill throughput 25x and decouple long-context cost from length. It also lists NVIDIA SM100 and Huawei Ascend 950 as equal first-class citizens, offering the world's AI infrastructure a third path around export controls. However, the KV cache, disaggregated deployment, and cross-instance load balancing remain closed source, it supports only a single model, and all performance figures are company self-tests—the standards war has only just begun.