DeepInfer Goes Open Source: $330K Annual Revenue Per GPU — How a Inference Framework Is Disrupting AI Compute Economics

As GPU prices continue to climb, the compute cost of AI inference has become an industry focal point. DeepSea has open-sourced its in-house inference framework DeepInfer and, for the first time, disclosed per-GPU revenue data — a conservative estimate of $330,000 in annual revenue per GPU, with gross margins approaching 97%. These numbers not only reveal the lucrative potential of AI inference but also point to an engineering revolution centered on KV Cache.

GPU Market: Compute Anxiety Amid Rising Prices

According to Bilibili's GPU Daily, prices for NVIDIA's RTX 50 series have hit all-time highs across the board: the RTX 5080 has risen to ¥16,849, even surpassing the RTX 5090's suggested retail price; the RTX 5070 Ti, 5070, and 5060 Ti have also climbed to ¥11,188, ¥7,349, and ¥3,849 respectively. Price trend charts show that current prices are still being suppressed by platform subsidies — otherwise mid- and low-end cards could be another ¥100–200 higher.

Memory prices have dipped slightly, but floor prices remain unchanged, trending sideways overall; CPU tray prices show mixed movements with floor prices equally stable. In the secondhand market, Xianyu transaction prices have been flat for a week, with speculation that the platform may no longer disclose data. The continued rise in GPU prices reflects the intense demand for AI compute — and DeepInfer's release serves as a high-efficiency footnote to that demand.

DeepInfer Open Source: Laying Out Cluster Deployment and Revenue Books

DeepSea recently open-sourced DeepInfer and published "DeepSea v4.1 Flash Large-Scale Inference Deployment Practice," detailing its cluster deployment scheme — including per-GPU hourly revenue data.

According to the article, even if everything were billed at stock price, per-GPU hourly revenue is $38, yielding a conservative annual revenue of $330,000. But DeepSea couldn't possibly sell all tokens at stock price — token pricing is twice the stock price. Outside observers estimate the average per-GPU hourly revenue may be around $46, corresponding to roughly $400,000 annually, or about ¥2.67 million. This may explain why "Big Fat Fish" is snapping up GPUs everywhere.

DeepSea's holding cost per GPU is not public, but third-party estimates put DeepInfer's gross margin at roughly 87.76% to 96.94%. Such margins are explosive by the standards of both hardware leasing and cloud computing.

Architecture Breakdown: Prefill/Decode Disaggregation and KV Cache Offloading to Disk

DeepInfer's core innovation is splitting servers into two parts: one dedicated to prefill and another dedicated to decode. This disaggregated architecture is optimized for the special workload of agent-style applications — agents repeatedly call tools and context, causing redundant prefill during inference and severely hurting efficiency.

DeepSea's solution is to place KV Cache on distributed storage: agent traffic is retained for three days, chat traffic for seven days. When the next request's "billing tag" matches, results are read directly from disk, skipping recomputation. The repeated prefill process becomes a disk read — roughly 96% of the input that would otherwise be recomputed is skipped outright, dramatically lowering inference cost.

The key to this design is binding the "billing tag" to cache-hit logic — only paid requests can hit the cache, ensuring a commercial closed loop while preventing cache abuse. From an engineering perspective, this is a textbook case of introducing a storage tier into the inference pipeline, trading distributed storage for GPU compute time.

KV Cache is a core optimization mechanism in large model inference. During autoregressive generation, Transformer must recompute attention over all previous tokens for each new token generated, and attention computation depends on the Key and Value vectors of each token. KV Cache stores these precomputed K and V tensors layer by layer in GPU memory (VRAM) for direct reuse during subsequent decoding, avoiding redundant computation and reducing the computational complexity of the generation phase from quadratic in sequence length to linear. The cost is that the cache occupies large amounts of VRAM — the longer the sequence and the higher the concurrency, the greater the memory pressure. DeepInfer's move to push KV Cache from VRAM down to distributed storage essentially transforms the "VRAM wall" problem into a storage bandwidth problem, trading larger-capacity, lower-cost storage for GPU compute time.

Prefill/decode disaggregation stems from the differing computational characteristics of the two phases. The prefill phase processes the entire input prompt at once — it is compute-intensive with high GPU utilization; the decode phase generates token by token — it is memory-access-intensive with much of the compute left idle. Running the two together causes mutual interference, leading to latency jitter and resource waste. Splitting them onto separate servers with independent scheduling allows hardware and concurrency strategies to be configured for each workload separately — one of the mainstream optimization directions in current large-model inference systems, alongside approaches such as vLLM's disaggregated prefill.

Versus Mining: A Compute Business with Higher Revenue Certainty

Under this inference framework, per-GPU revenue stability far exceeds that of crypto mining. Mining revenue is affected by coin price volatility, difficulty adjustments, and electricity costs, whereas inference service revenue has far greater certainty — demand comes from real AI application calls, and DeepInfer further lowers marginal cost through cache reuse.

For DeepSea, the biggest risk today may not be insufficient demand but being unable to buy enough GPUs to meet it. The GPU price surge echoes the inference framework's high margins: the compute supply bottleneck is becoming the core competitive moat for AI inference service providers.

DeepInfer's open-sourcing gives outsiders a clear first look at the economics of large-scale inference deployment. For teams looking to build their own inference clusters, this prefill/decode disaggregation plus KV Cache disk-offloading architecture offers a reusable path to cost reduction; for the AI industry as a whole, the $330,000-per-GPU annual figure may redefine the valuation logic of "compute as revenue."

One-line Summary

DeepSea open-sources its DeepInfer inference framework — $330K annual revenue per GPU, gross margins up to 97%, disrupting compute economics via KV Cache disk offloading.

Paragraph Summary

DeepSea has open-sourced its in-house inference framework DeepInfer, disclosing large-scale cluster deployment and per-GPU revenue data: $38 hourly revenue per GPU, a conservative annual estimate of $330,000, with actual token pricing implying an average of up to $400,000, and gross margins between 87.76% and 96.94% — far above hardware leasing and cloud computing. Its core innovation is disaggregating prefill and decode deployment and pushing KV Cache down to distributed storage, retaining agent traffic for three days and chat traffic for seven days, binding cache hits to billing tags, so that approximately 96% of redundant prefill input can be skipped, greatly reducing inference costs. The article also notes that RTX 50 series GPU prices have hit new highs across the board, the compute supply bottleneck is becoming the core moat for inference providers, and per-GPU revenue certainty clearly exceeds that of mining.