DeepSeek Open-Sources DeepInfer: Inference Costs Slashed and Commercial Licensing Fully Open

DeepSeek^Ad has open-sourced the inference framework used for its own online business. The project, called DeepInfer, draws attention not for its technical details but for two things: a license so permissive that it can be used directly for commercial purposes, and an engineering approach that pushes inference costs to the extreme. Inference is currently the most expensive part of large models, and DeepSeek's choice to attack it here sends a clear signal—the inference cost war has already begun.

DeepInfer Open Source: A Framework That Has Served Its Own Business

DeepInfer originally served only DeepSeek's online business; it was not a toy from the lab. This means it has been through real traffic, real latency requirements, and real cost pressure—not just pretty numbers on benchmarks. The open-source version continues the permissive license, allowing developers to use it directly in commercial products without having to navigate licensing terms.

For small and medium-sized teams, this is more attractive than performance metrics. When choosing an inference framework, teams often get stuck on licensing, especially in scenarios where it needs to be embedded in commercial products. DeepSeek has removed this barrier entirely, effectively winning a victory at the ecosystem level: the more people use it, the richer the engineering practices and tuning experience around DeepInfer become, which in turn lowers the cost of subsequent use.

Speculative Decoding: Let the Model Guess First, Then the Large Model Verifies

DeepInfer's core weapon for saving money is speculative decoding. The idea is not complicated: first use a small model or lightweight strategy to guess the next few words, then have the large model verify them all at once. If the guess is correct, accept it directly; if wrong, fall back. The most expensive part of a large model is a single forward pass. If one verification can replace multiple token-by-token generations, throughput naturally goes up.

The difficulty of this method lies in "guessing accurately" and "verifying without waste." Guessing too conservatively yields no benefit; guessing too aggressively means the fallback cost eats the profit. DeepInfer's value lies in engineering this mechanism to the point where it can go live, and validating it with real business data. According to calculations in the video, after adopting this solution, the payback period for inference services shrinks from 10 months to 4 months. If placed in the context of a self-built inference cluster, this number means the recovery speed of capital expenditure more than doubles.

It should be noted that speculative decoding is not a DeepSeek invention; there are related papers and implementations in both academia and industry. DeepInfer's differentiator is that it was polished in tandem with online business and open-sourced together with the framework. Looking at the technical route alone, it is not revolutionary, but the combination of "open source + commercially usable + business-validated" lowers the barrier significantly for teams wanting to build their own inference services.

Key Metrics for Speculative Decoding: Acceptance Rate and Speedup Ratio

The actual benefit of speculative decoding depends on two quantities: the draft model's acceptance rate and the number of tokens a single verification can cover. Ideally, if the draft model guesses k tokens, the large model can verify these k positions in parallel in one forward pass. The higher the acceptance rate and the larger k, the closer the theoretical speedup ratio is to k. But the acceptance rate decreases as k increases, and the draft model itself has inference overhead, so there is an optimal value of k. In engineering, a common practice is to dynamically adjust the draft length and draft model size based on online traffic—which is why frameworks like DeepInfer need to be refined with real business data; offline benchmarks can hardly reproduce the distribution of real requests.

Not Just DeepSeek: The Inference Efficiency Race Is Opening on Multiple Fronts

There are two other pieces of information worth considering together. One is that Anthropic is preparing a "guess intent" model that, when fixing a bug, can identify an entire class of problems rather than just plugging a single vulnerability. The other is Inception's Mercury3, which is said to output 1,000 tokens per second.

All three pieces of information point in the same direction: competition at the inference layer has shifted from "how big is the model" to "how cheap and how fast is each token." DeepSeek uses speculative decoding to cut costs, Inception uses throughput to grab scenarios, and Anthropic tries to make the model think clearly in one pass, reducing waste from repeated calls. Different routes, but all answering the same question—how to make inference more cost-effective.

For developers, this means the metrics to look at when choosing a solution have changed. Looking only at model capability leaderboards is not enough; you also need to look at cost per token, first-token latency, batch throughput, and optimization space at the framework level. After DeepInfer is open-sourced, at least one of these metrics can be tuned by yourself.

Why First-Token Latency and Batch Throughput Are Two Separate Metrics

Inference service performance cannot be judged by a single number. First-token latency (TTFT) measures the time from when a user sends a request to when they see the first character, determining the interactive experience; batch throughput (tokens/s per GPU or per cluster) measures how many tokens a unit of hardware can produce per unit time, determining unit cost. The two often pull against each other: increasing batch size can improve throughput but raises first-token latency. Speculative decoding mainly improves throughput and cost per token, with limited improvement to first-token latency—it may even increase slightly due to the draft phase. Therefore, when evaluating an inference framework, distinguish which type of metric it optimizes, then compare it with your own scenario—real-time conversation values latency, while offline batch processing values throughput.

Another Thread: Certificate Leaks and Python Tool Updates

In the same batch of information, there is also a security-related item: Emodera leaked the private keys of 8 intermediate certificates, and these certificates are trusted by default by systems and browsers. A leaked intermediate certificate private key means an attacker could potentially sign certificates that are widely trusted. The risk is not about a particular website but about the trust chain itself.

Several related tools in the Python ecosystem have released new versions, and the usual handling is to update the certificate revocation list and trust store. If your service relies on Python's certificate verification chain, it is worth checking dependency versions. This item is not directly related to inference frameworks, but it reminds us of one thing: cost optimization and security maintenance at the infrastructure layer often happen simultaneously. The money saved by open-source frameworks may end up being spent on dependency governance.

Why an Intermediate Certificate Private Key Leak Is More Dangerous Than a Single-Site Leak

In the TLS trust chain, an intermediate certificate (intermediate CA) is issued by a root certificate and has the authority to continue issuing end-entity certificates. Browsers and operating systems trust a set of intermediate certificates by default. Therefore, once the private key of an intermediate certificate leaks, an attacker can use it to sign certificates for any domain that are widely trusted, without needing to compromise the root certificate. Such risks are difficult to eliminate completely through blacklists. The standard response is for the CA to add the leaked intermediate certificate to a revocation list (CRL) or mark it as revoked via OCSP, while relying parties update their trust stores. For developers, this means the CA bundles and components such as Python's certifi that the certificate verification chain depends on need to be kept up to date; otherwise, revoked certificates may continue to be trusted.

The Real Pain Point for Small Businesses: No One Answers the Phone

The last item in the information source comes from small business owners: their biggest pain is not that the model is not powerful enough, but that no one answers the phone after hours. This was ranked highest in the day's information because it has clear willingness to pay and clear scenario boundaries.

Looking at it together with DeepInfer, they are actually two sides of the same coin. Lower inference costs benefit not only big companies but also small scenarios that previously could not afford real-time AI services—such as after-hours phone answering and non-business-hours customer service responses. Every time cost drops by a notch, a batch of scenarios that previously did not add up becomes feasible. The long-term impact of DeepSeek open-sourcing its inference framework may not be about how much money it saves, but about which new scenarios become cost-effective for the first time.

The inference cost war has just begun. The real winner may not be the one with the most parameters, but the one that brings unit costs down first.

One-line Summary

DeepSeek open-sources its online inference framework DeepInfer, using speculative decoding to cut inference costs to a 4-month payback period and opening up commercial use.

Paragraph Summary

DeepSeek has open-sourced DeepInfer, the inference framework serving its own online business. The license is permissive and allows direct commercial use, making the barrier extremely low for small and medium-sized teams that need to embed it in commercial products. Its core optimization is speculative decoding: a small model guesses several tokens first, and the large model verifies them in one pass. Calculations show the payback period shrinks from 10 months to 4 months. The article groups this move with Anthropic's intent-guessing model and Inception Mercury3's 1,000 tokens per second, pointing out that inference competition has shifted from model scale to cost per token and speed, and that developers' selection metrics are changing accordingly. It also mentions Emodera's leak of 8 intermediate certificate private keys that are trusted by default and Python tool updates, and uses the example of small businesses with no one answering the phone after hours to show that lower inference costs will make real-time AI scenarios that previously did not add up economically feasible for the first time.