DeepSeek Open-Sources Inference Engine: The Inside Story of Tripling GPU Value

DeepSeek released the updated version v4.1 Flash 10.17 (hereafter 10.17) just 37 days after launching v4.1 Flash, with an average benchmark improvement of 13.5%. More importantly, they open-sourced DeepInfer, the core engine powering their production inference. According to an in-depth analysis by Bilibili creator "bycloud," this engine boosts revenue per GPU-hour from the previous $12.91 to $38—nearly triple. This means that open-weight models combined with an extremely optimized inference stack are rewriting the economics of AI services.

The Leap in Inference Economics: From $12.91 to $38

DeepSeek^Ad's published inference economics data is striking: during off-peak hours, each GPU can earn $38 per hour. Compare this to the V3/R1 deployment data published in February 2025—at that time, 226.75 servers (each with 8 H800 GPUs, totaling about 1,814 GPUs) processed 776 billion tokens per day, generating approximately $562,000 in daily revenue at R1 API prices, equivalent to $12.91 per GPU-hour. Now the 10.17 model performs better yet earns nearly three times as much, and the key contributor is the aggressive optimization of the inference stack.

Pre-fill and Decoding: Two Workloads, Two Strategies

LLM inference consists of two stages: pre-fill, which processes the input context and is compute-intensive, and decoding, which generates tokens one by one and is constrained by memory bandwidth. DeepSeek deploys the two separately, each with a different parallelism strategy. Pre-fill uses small expert-parallel groups of 2-8 GPUs to avoid excessive communication overhead. Benchmarks show that when the batch size increases from about 4,000 tokens to 16,000 tokens, throughput rises from about 60,000 tokens/s to 82,000 tokens/s, but per-step latency increases from about 70 ms to 200 ms; increasing further to 65,000 tokens raises throughput only slightly to 90,000 tokens/s while latency exceeds 700 ms. Diminishing returns are obvious, so pre-fill does not blindly scale up GPUs.

For decoding, they adopt wide expert parallelism (Wide EP), distributing the load across 64-128 GPUs. Under EP64, each GPU only needs to hold about 7 experts (about 52 under EP8), greatly reducing memory reads. In actual tests at 240 tokens/s per user, the 8-GPU setup achieves only 1,900 tokens/s per GPU, while the 64-GPU setup jumps to 157,000 tokens/s—an over 8x improvement. But if the per-user speed requirement is relaxed to about 1 second, the gap between the two shrinks to 6%—the advantage of wide parallelism only stands out in extreme low-latency scenarios.

Expert Parallelism (EP) is a key strategy for distributed inference of Mixture-of-Experts (MoE) models. MoE models contain multiple "expert" sub-networks, and each token activates only a few of them. EP distributes these experts across different GPUs, with each GPU responsible for computing a subset of experts. A larger EP value (e.g., EP64) means each GPU needs to store and compute fewer experts, reducing memory pressure, but the communication overhead for routing tokens to other GPUs also increases. DeepSeek uses small-scale EP (2-8 GPUs) in the pre-fill stage because pre-fill is compute-intensive with long sequences, requiring priority control of communication latency; in the decoding stage, it uses wide EP (64-128 GPUs) because decoding is memory-bandwidth-bound, and distributing experts significantly reduces the weight reads per GPU, thereby improving throughput. This differentiated design of EP scale based on workload characteristics is one of the core ideas behind DeepInfer's efficient inference.

KV Cache: 96% Hit Rate and 100 PiB Storage

In agent workloads, a large number of input tokens are repeated. DeepSeek's inference system reports that 56.3% of input tokens are served via disk cache, and in the latest deployment the cache hit rate is as high as 96%. To this end, they built their own distributed file system 3FS, dedicating about 100 PiB of SSD exclusively to KV cache, equivalent to 14.4 TB of logical cache per online GPU, retained for up to 3 days (agent traffic) to 7 days (chat traffic). Combined with caching, per-GPU input throughput jumps from 60,000 tokens/s to about 1.5 million tokens/s—a 25x logical improvement, even though the GPU itself is not 25 times faster.

KV Cache is a key mechanism in autoregressive generation to avoid redundant computation. During Transformer decoding, the attention computation for each new token requires the Key and Value vectors of all previous tokens. Without caching, generating each token requires recomputing the K/V for the entire prefix, with computation growing quadratically with sequence length. KV cache stores the already-computed K/V, so a new token only needs to compute its own and concatenate with the cache, reducing complexity to linear. However, KV cache size grows linearly with context length and the number of concurrent requests, placing enormous pressure on GPU memory and storage. In DeepSeek's agent workloads, a large number of input tokens are repeated, so KV cache is persisted to distributed storage (3FS) and achieves a hit rate of up to 96%, eliminating recomputation of repeated prefixes and greatly improving effective throughput. The 100 PiB of SSD dedicated to caching provides 14.4 TB of logical cache space per online GPU, which is the material basis for achieving a 25x logical input throughput improvement.

What Was Open-Sourced and What Wasn't

DeepInfer open-sources the optimized inference engine, kernels, expert parallelism, DSpark speculative decoding, and other core components, but the distributed KV cache system, request load balancer, and coordination infrastructure across pre-fill/decoding replicas are not included. Currently, it only supports offline batch inference; using it for production serving still requires substantial self-built infrastructure. On the hardware side, it supports NVIDIA SM100 (such as B200) and Huawei Ascend 950, and recommends at least 64 GPUs for pre-fill and another 64 for decoding—that is already about $9 million worth of hardware, costing about $1,024 per hour (excluding storage).

Speculative Decoding is a technique to accelerate autoregressive generation. It uses a lightweight "draft model" to quickly generate multiple candidate tokens, and then the target large model verifies these candidates in parallel, keeping the accepted ones. Since the large model can verify multiple tokens in a single forward pass and the draft model's generation cost is very low, overall decoding speed can improve 2-3x while the output distribution remains exactly the same as the target model (lossless acceleration). The DSpark speculative decoding component open-sourced by DeepSeek is one of the key optimizations in its decoding stage for reducing latency and improving per-GPU throughput. The effectiveness of speculative decoding depends on the consistency between the draft and target models and the acceptance rate; in scenarios such as agents and chat, combined with the high KV cache hit rate, it can further amplify the speedup.

Revenue Estimates and Gross Margin

Based on revenue of $38 per GPU-hour, assuming a deployment of about 6,000-8,000 GPUs (inferred from the ratio of 100 PiB storage to 14.4 TB cache per GPU), annualized API revenue could reach $2-2.7 billion (at off-peak prices only). On the cost side, at a rental price of $3-6 per GPU-hour, serving 1 billion billable tokens incurs GPU costs of about $1.15-2.29, while revenue is $14.79, yielding a gross margin of about 84%-92%. Even with self-purchased GPUs, such a profit margin is extremely rare in compute-heavy services.

Some researchers compared DeepInfer's performance with SemiAnalysis's InferenceX benchmark: at 100 tokens/s per user, DeepInfer achieves about 26,000 tokens/s output throughput per GPU, 4.3 times that of the best configuration in the benchmark (about 6,000 tokens/s). Open-sourcing this engine means other teams can reproduce these optimizations, although it currently only adapts to the DeepSeek v4.1 Flash architecture, its kernels and ideas can be studied and borrowed.

Gross margin is a core metric for measuring the economics of AI inference services, calculated as (revenue - direct cost) / revenue. Here, direct costs mainly include GPU rental or depreciation, electricity, storage, and networking, excluding R&D and labor. The 84%-92% gross margin estimated by DeepSeek is striking because compute-heavy services typically have lower margins due to expensive GPUs and fluctuating utilization. The high margin comes from three sources: first, inference stack optimization multiplies per-GPU throughput, diluting the GPU time per token; second, KV cache shifts the computational cost of a large number of repeated inputs to cheap SSD storage; third, off-peak pricing fills electricity valleys. Note that this estimate is based on full-load operation and off-peak prices; actual production is also affected by request fluctuations, free quotas, peak pricing differences, and other factors, but even so, its cost structure is still significantly better than the industry average.

Conclusion: Another Victory for Open Weights

By open-sourcing its inference engine, DeepSeek makes its extreme inference optimizations public. This not only lowers the barrier to high-performance AI services but also demonstrates to the industry the enormous potential of open-weight models combined with deep optimization. Of course, to truly reproduce its production-grade deployment, a great deal of infrastructure still needs to be filled in. But at least, the value of GPUs has been redefined—not because the hardware became stronger, but because software enables each GPU to deliver multiple times the performance.

One-line Summary

DeepSeek open-sources its inference engine DeepInfer, nearly tripling revenue per GPU-hour and revealing how software optimization reshapes the economics of AI services.

Paragraph Summary

Along with the release of v4.1 Flash 10.17, DeepSeek open-sourced its production-grade inference engine DeepInfer. Through separate deployment of pre-fill and decoding, differentiated expert parallelism, and its self-developed 3FS distributed storage achieving a 96% KV cache hit rate, the engine boosts revenue per GPU-hour from $12.91 to $38—nearly triple. Benchmarks show per-GPU output throughput of 26,000 tokens/s, 4.3 times the best existing configuration. Although the open-source release does not include production components such as distributed KV cache and load balancing, and currently only supports offline batch processing, its kernels and optimization ideas provide an important reference for the industry. API services based on this engine are estimated to generate annualized revenue of $2-2.7 billion with a gross margin of about 84%-92%, demonstrating the enormous economic potential of open-weight models combined with extreme inference optimization.