DeepSeek's Open-Source Inference Framework: The Compute Economics Behind $330K Annual Revenue Per GPU

Title

DeepSeek's Open-Source Inference Framework: The Compute Economics Behind $330K Annual Revenue Per GPU

Meta Description

DeepSeek^Ad open-sourced an inference framework. How do you account for $330K annual revenue per GPU and $18M RMB/year for an 8-GPU machine? A breakdown of prefill-decode disaggregation, the KV cache pool, and the API-compatible repo—plus the three strategic intents: freeing up compute, countering bans, and showing cards to investors.

Article Content

DeepSeek open-sourced an inference framework, and the accompanying documentation spells out the math in black and white: one B300 GPU running at full capacity for a year brings in $38/hour × 24 hours × 365 days ≈ $330K. Eight GPUs make one machine, generating $2.66M in annual revenue—over 18 million RMB. As soon as the news broke, Bilibili creator "Mengmeng" crunched the numbers and joked, "It's not that I don't want to earn this money—it's that I'd first need 8 B300s." Jokes aside, what's really worth examining about this framework is that it lifts the curtain on the cost structure of inference serving, along with DeepSeek's strategic intentions across three fronts: compute, ecosystem, and capital.

$330K Annual Revenue Per GPU: How the Math Works

When DeepSeek open-sourced the inference framework on October 20, the accompanying technical Q&A unusually provided a "floor price": a minimum revenue of $38 per GPU per hour. Cloud providers typically only report total revenue, model companies only report price per million tokens—nobody puts per-GPU hourly output on the table. DeepSeek not only said it but labeled it a "floor price"—meaning don't calculate with peak prices; this is the minimum you can expect to collect.

By this measure, an 8-GPU machine generates $2.66M in annual revenue against a cost of roughly 3.5 million RMB, making a six-month payback sound "perfectly reasonable." An even more intuitive comparison comes from processing capacity: a single cluster processes 155 trillion tokens per day—equivalent to what over 40 million people would chat through in an entire year, done in one day.

These numbers aren't pulled from thin air. In the inference system documentation DeepSeek published in February 2025, they had calculated costs: an average of 226.75 nodes running, rented at $2/GPU/hour, costing $87K per day; priced at R1's rates at the time, revenue was $562K per day, yielding a cost-profit margin of 545%. Last year they calculated "how much I spent"; this year they're reporting "how much I collect"—what lies in between is how much work one GPU can do and how much money it can generate.

Technical Breakdown: Prefill-Decode Disaggregation, KV Cache Pool, and Speculative Decoding

The framework's most core change is separating "reading" from "writing." When a user sends a message, the model first has to read through the question—this step is bandwidth-intensive; then it spits out one character at a time—this step is memory-intensive. Cramming both onto one machine leaves neither side happy. DeepSeek's solution: use ordinary 8-GPU machines for reading, and large 64- or 128-GPU machines for writing. The minimum configuration is 64 GPUs for reading and 64 for writing—128 GPUs total—and reading and writing can even be placed in different data centers connected by fiber optics. The trade-off: if a construction crew accidentally cuts the fiber, you're in for a long night; the benefit: faster time-to-first-token and steadier output—users swipe away if a short video doesn't load in two seconds, and they'll leave just as quickly if the AI's first character is slow.

The second piece is KV cache. During a conversation, the model needs to remember what was said earlier, but memory costs both storage and compute. Paste in a contract, then ask about the same contract the next day—as long as it remembers, it doesn't need to re-read tens of thousands of characters. DeepSeek built a pool with 100,000 TB capacity using a distributed file system, retaining agent memory for three days and human chat for seven days—old conversations have long-tail memory, and a small subset of content gets pulled up repeatedly; losing it drops the hit rate, so keeping it longer is saving costs for users.

The third is "guessing characters": let a small model guess a few candidates first, then have the large model pick the one it's most confident about. The verification volume per step drops by 30%, and throughput increases by 4% to 7%.

They also open-sourced a small repo for API interface conversion, whose purpose is to make DeepSeek endpoints deployed by others behave exactly like the official API. This solves the previous pain point where "the official site is smooth, but deployments elsewhere are slow, dumb, and drooling."

The computational differences between Prefill and Decode are the prerequisite for understanding prefill-decode disaggregation. In the Prefill phase, the model processes the entire input in parallel at once—matrix operations are dense, compute utilization is high, but it requires instantly moving large amounts of weights and intermediate activations, making it bandwidth-bound. In the Decode phase, each step generates only one token but must repeatedly read all model weights and KV cache—this is memory-bandwidth-bound, leaving GPU compute largely idle. Running both workloads together causes them to compete for resources, and long Prefill requests also slow down time-to-first-token for co-batched Decode requests. The industry's common Chunked Prefill and PD Disaggregation (Prefill-Decode Disaggregation) approaches are designed for this; DeepSeek's solution is a large-scale engineering implementation of the latter.

KV cache refers to caching the Key and Value tensors from each layer's attention mechanism during autoregressive inference, so already-processed context doesn't need to be recomputed. For a 70B-class model in long-context scenarios, a single request's KV cache can reach several GB; with high concurrency, GPU memory fills up quickly, directly limiting the number of users that can be served. So KV cache is both a memory cost and a hit-rate asset: a cache hit skips the Prefill computation for the corresponding prefix. Industry approaches like Prefix Caching and PagedAttention all optimize around this, while DeepSeek sinks the cache down to a distributed storage pool and sets retention periods by scenario—essentially trading off between storage cost and recomputation cost.

Industry Reaction: The Black Box Is Open, and Inference Premiums Can't Hold

Frameworks used to be black boxes—you knew it was slow, but not where. Now the code is right there; wherever it falls short, you can compare line by line. The ones most uncomfortable are inference service vendors—previously customers didn't understand and could only buy off-the-shelf; now the whole thing is public, and customers can do the math themselves.

Bilibili creator "Mengmeng"'s judgment: AI services will drop further in price. Inference cost is the most fiercely fought battle among domestic players—whoever has lower costs dares to undercut, and the prices of phone assistants, photo editors, and translation software are all squeezed at this layer. This open-sourcing means subsequent software is very likely to get even cheaper.

Strategic Intent: Free Up Compute, Do Business, Show Cards to Investors

There are three layers of consideration behind the open-sourcing.

First, free up compute for themselves. That small API interface repo solves the "inconsistent experience of external deployments" problem—users no longer crowd onto the official site, so the official site's GPUs can be moved to training next-generation models and pursuing AGI. With domestic compute tight and multiple parties competing for GPUs, tossing the service burden outward is a real reduction in load.

Second, a business calculation. On October 19, the U.S. Department of Homeland Security ordered federal civilian agencies to purge DeepSeek products, including third-party tools, and barred new federal contracts; immediately afterward, on October 20, this framework was open-sourced. The two events are adjacent; officials didn't state a causal link, but the creator's view is direct: you ban yours, I'll open-source mine—you can control institutional procurement, but you can't control line-by-line code downloads.

Third, a strategic signal for investors. In June, DeepSeek completed its first external funding round—over 50 billion in scale at a $50B valuation, with Liang Wenfeng personally contributing 20 billion for a 40% stake; in October, word came of a second round increased to 100 billion, with CICC providing IPO guidance for the STAR Market; in September, they recruited a CFO. Money's in place, people are in place, the channel is in place—at this critical juncture, open-sourcing the "engine that puts food on the table" is about letting people see: I can give this away because I have something better in hand—I can still build next-generation models and AGI. Liang Wenfeng holds over 80% of the shares, so the company's value is tied to his personal net worth; what investors look at is whether there are still cards to play tomorrow.

Speculative Decoding is the academic name for the "guessing characters" step: use a small draft model to autoregressively generate several candidate tokens first, then have the large model verify them in parallel in a single forward pass—accepted tokens are used directly, rejected ones are resampled from the first error position. Since verification is parallel computation, the large model's actual forward passes are reduced, while the output distribution remains mathematically identical to pure large-model sampling (greedy or sampling by corrected acceptance probabilities). The gains depend on the draft model's acceptance rate and the cost ratio between the two models; common speedups range from 1.5× to 3×. DeepSeek's report of a 30% reduction in verification volume and 4% to 7% throughput improvement is a relatively conservative engineering gain.

Editorial View

The "floor price" of $38 per GPU per hour is the most noteworthy signal from this framework—it lays bare the cost floor of the inference business to everyone, and pushes competition from "selling services" toward "competing on efficiency." Prefill-decode disaggregation and the KV cache pool aren't showing off—they're real optimizations on time-to-first-token and long-conversation costs, and these engineering details will ultimately pass through to end-user software prices. As for the timing of the open-sourcing versus the ban order, officials haven't stated a connection, but one observable fact is: once code is public, it's easy to block institutional procurement but hard to block technology diffusion. For ordinary users, the endgame of this compute economics is most likely cheaper AI services.

One-line Summary

DeepSeek open-sourced its inference framework, revealing a $38/hour/GPU cost floor that compresses industry premiums.

Paragraph Summary

DeepSeek open-sourced an inference framework on October 20, with documentation unusually providing a floor revenue price of $38 per GPU per hour—an 8-GPU machine generating about $2.66M annually (over 18 million RMB) against roughly 3.5 million RMB in costs. The article breaks down its technical core: prefill-decode disaggregation (8-GPU machines for Prefill, 64- or 128-GPU machines for Decode, across data centers), a 100,000 TB distributed KV cache pool (3-day retention for agent memory, 7 days for chat), small-model speculation cutting verification volume by 30% and boosting throughput 4%–7%, and an API compatibility repo ensuring external deployments match the official API. Open-sourcing opens the inference black box, squeezing premium space for inference service vendors and promising further price drops in AI applications. Strategically, the open-sourcing serves three intents: freeing compute for training, countering the U.S. federal agency purge order, and signaling to investors that better cards remain in hand.