DeepSeek Open-Sources Its Inference Framework DeepInfer: The Economics of Inference Services and Industry Impact

DeepSeek has open-sourced its in-house inference framework DeepInfer, sparking heated discussion in the tech community. This is not just a technical share — it's the first public disclosure of the cost-revenue model for inference services: a 128-GPU cluster generating over $40 million in annual revenue. This article breaks down the technical details of the inference framework, the real intent behind open-sourcing it, and which companies will truly benefit.

What Is an Inference Framework and Why It Matters So Much

A large model is fundamentally Next Token Prediction; the process from user input to output return is called Inference. Serving a single user requires just one GPU, but handling massive user requests demands GPU cluster scheduling engineering capability — that's what an inference framework is.

The core value of an inference framework lies in cutting costs and boosting efficiency. DeepSeek^Ad's framework is deeply optimized for its own models, using multiple technical means to drive down costs and improve output efficiency.

DeepInfer's Core Technical Optimizations

KV Cache: Avoiding Redundant Computation

When multiple users submit the same content (say, the same report), the model caches the already-computed portions and directly reuses cached results for identical inputs thereafter. According to the video share, DeepSeek's cache-hit cost is $0.003 per million tokens versus $0.15 per million tokens on a miss — a 50x difference. Output cost is about $0.6 per million tokens. This caching mechanism dramatically cuts redundant computation expenses.

KV Cache (Key-Value Cache) is a core optimization technique in autoregressive generation. During Transformer decoding, each new token generated requires attention computation over the Key and Value vectors of all previous tokens. Without caching, these intermediate results would be recomputed repeatedly, with computation growing quadratically with sequence length. KV Cache stores the computed Key/Value tensors in GPU memory, so a new token only needs to compute its own and concatenate with the cache, reducing complexity to linear. The cost is memory usage: cache size is proportional to sequence length, number of layers, and number of attention heads. Production environments also need to handle cache eviction, prefix sharing across requests (which is exactly where DeepSeek's cache-hit pricing comes from), and memory fragmentation management — a high degree of engineering complexity.

Prefill/Decode Disaggregation: Improving GPU Utilization

The input (Prefill) and output (Decode) phases have different computational characteristics. Splitting them onto different GPUs enables more efficient use of hardware resources. This idea also appears in open-source frameworks like vLLM, but DeepSeek has done more fine-grained adaptation for its own models.

The Prefill phase processes the entire input sequence in parallel and is compute-bound, fully utilizing GPU compute; the Decode phase generates token by token, computing only one word per step, and is memory-bound, leaving much GPU compute idle. If both run on the same GPU, Decode gets blocked by Prefill's long computations, causing output latency jitter. Disaggregated deployment runs the two workloads on different GPU pools, each scheduled with optimal batching strategies, improving overall throughput while stabilizing first-token latency. This architecture has been explored in systems like vLLM's disaggregated prefilling and Splitwise, but it requires solving the overhead of transferring KV Cache across nodes between GPUs, placing high demands on interconnect bandwidth.

Speculative Decoding and Heterogeneous Scheduling

Speculative decoding lets the model "guess" upcoming words first; if correct, it skips token-by-token generation, improving parallelism. The framework also handles scheduling between GPUs (e.g., a 128-GPU cluster) and between GPU and CPU: when a flood of requests arrives, the CPU queues them properly, distinguishing idle from busy periods to avoid GPUs sitting idle.

These optimizations yield limited gains individually, but stacked together the improvement in overall efficiency is considerable.

Speculative Decoding uses a lightweight "draft model" to quickly generate several candidate tokens, which the target large model then verifies in parallel in a single pass. If verification passes, the candidates are accepted directly — effectively replacing multiple serial generation steps with one large-model forward pass, speeding up decoding without changing the output distribution. The draft model can be a smaller model, certain layers of the model itself, or retrieval-based n-gram guesses. The benefit depends on the draft hit rate: the higher the hit rate, the more noticeable the speedup; if draft quality is poor, failed verification actually adds overhead. Industrial deployment also requires handling dynamic draft length adjustment and compatibility with continuous batching.

First Public Disclosure of the Economics of Inference Services

DeepSeek calculates based on "off-peak pricing" (low-traffic periods such as Sundays): roughly $38 in revenue per GPU per hour. Over 365 days a year, 24 hours a day, on a 128-GPU cluster, that's about $40+ million in annual revenue (over 300 million RMB). Enterprises can use this to assess whether buying 128 GPUs can deliver sustained and stable ROI.

This is the first time DeepSeek has shown the world that inference services can make money this way — that providing model inference itself can generate ongoing revenue.

Who Really Benefits?

Indirect beneficiaries: SMEs and manufacturing. They won't buy their own GPUs (unavailable or too expensive) but benefit from continuously falling API prices.

Direct beneficiaries: inference service providers. Cloud providers like Tencent Cloud, Alibaba Cloud, and ByteDance, as well as third-party API providers like SiliconFlow, can directly use this open-source framework to improve efficiency and reduce costs.

But open source doesn't mean "plug and play." In offline environments you can test the framework and learn the technical details, but online deployment still requires substantial engineering capability: the KV Cache mechanism, cross-cluster load balancing, and the specific implementation of Prefill/Decode disaggregation are not fully disclosed. Users of existing open-source frameworks like vLLM and SGLang can borrow its operators, ideas, and module optimizations to improve their own inference frameworks.

The Real Intent Behind Open-Sourcing: Short-Term Traffic Diversion, Long-Term Ecosystem Building

Some worry: once open-sourced, with third-party services rising, who will still use DeepSeek's own API? In the short term, some API purchases will indeed be diverted, but DeepSeek has not open-sourced all its online service details, and it will take time for other vendors to adapt.

The deeper value lies in: accelerating technology adoption and understanding, attracting open-source talent, and releasing capabilities for the industry to advance together. When more inference providers are willing to run DeepSeek models, the model's value becomes more evident — in an era flooded with models from vendor A, B, and C, efficiency, quality, and low cost are what make users choose. This is essentially about building an ecosystem and setting standards.

According to the video share, DeepSeek recently completed its second funding round and may go public in 2027. Open-sourcing the inference framework at this juncture reflects both a commitment to technology for the public good and a strategic move to secure ecosystem positioning.