DeepSeek Open-Sources Inference Engine: 80% Gross Margin Crushes Silicon Valley, but Copycats Find the Assembly Line Missing

Over the past year, one of the biggest mysteries in tech has been: how does DeepSeek manage to push model API prices so low? While Silicon Valley peers footed hefty cloud bills and watched DeepSeek price near-free intelligence, the common assumption was that this was a cash-burning bluff to buy market share. That assumption was shattered once DeepSeek published the real numbers from its data centers alongside its production software.

According to an analysis by a US tech blogger (reposted by a Bilibili creator), a single DeepSeek^Ad cluster processes 155 trillion tokens per day—roughly 1.8 billion words per second. Even at discounted rates, a single GPU generates about $38 in revenue per hour. Enterprise-grade AI chips typically cost $5 to $10 per hour to run, which implies DeepSeek's gross margin is estimated at a staggering 80%.

From "Impossible" to Open Source on GitHub: DeepInfer and DeepSeek Recipe

A year and a half ago, DeepSeek told the community that open-sourcing production-grade serving code was "impossible" because it was too tightly coupled to its private data centers. Now they've delivered on that promise, putting the inference engine that powers this system—DeepInfer—and its companion gateway, DeepSeek Recipe, on GitHub. Within 24 hours of release, the repo surpassed 9,000 stars, later approaching 20,000, as developers rushed in to see what was inside.

This is more than a code donation. It turns "low price" from a business-strategy question into an engineering-efficiency question: DeepSeek isn't sustaining prices through subsidies, but through architecture and scheduling that push per-token cost to a level others struggle to replicate.

Architectural Breakdown: Cutting the 40-Layer Pipeline in Half

When a traditional large model processes a long prompt, it must pass every word through all network layers before writing the first character—and this "reading phase" is extremely compute-intensive. DeepSeek uses a causal encoder-decoder design that cleanly splits a typical 40-layer pipeline in two: the first 20 layers act as a dedicated reader, and the last 20 as a writer. When a prompt comes in, only the first 20 layers run; the last 20 receive information via a mathematical projection, halving the reading workload on the spot.

The model itself is a Mixture-of-Experts architecture: 748 billion total parameters, but reading a prompt only wakes 80 billion, while generating an answer wakes 160 billion. This "wake on demand" approach directs compute where it's needed instead of letting the whole network spin idle.

Causal Encoder-Decoder is key to understanding this architecture. When a traditional Transformer decoder processes a prompt, each token must pass through all layers in sequence, and the generation phase reuses those layers' KV cache—so "reading" and "writing" share the same set of parameters and constrain each other. DeepSeek's design dedicates the first several layers purely as an encoder, responsible only for compressing the prompt into a set of representations; the later layers act as a decoder, receiving the encoded result directly via a mathematical projection (rather than layer-by-layer forward passes) and beginning autoregressive generation. This requires the encoder output and decoder input to be dimensionally aligned, with the projection matrix learned during training. The trade-off is that the encoder no longer participates in generation, so reading and writing parameters can be independently optimized for two different workloads—compute-intensive and bandwidth-intensive respectively. This is precisely the structural foundation that enables the wide expert parallelism described below to be split across deployments.

DeepInfer's Four Engineering Moves

Move 1: Wide Expert Parallelism—Handing 100 Books to 128 People

Reading and writing are two entirely different problems: reading is compute-bound, writing is memory-bandwidth-bound. For every word generated, the GPU must move billions of weight numbers from memory. DeepSeek's approach: use cheap 8-GPU servers for reading, while for writing, connect 64 to 128 GPUs into a supernode and spread 384 experts across 128 chips, each holding only about 7 experts. Instead of one person carrying 100 books, 128 people each take one or two—memory traffic drops by nearly 88%, and a single GPU can output over 26,000 words per second.

Mixture-of-Experts (MoE) is the underlying mechanism behind "wake on demand" in this article. A standard dense model passes every token through all parameters; MoE splits the feed-forward layer into many "expert" sub-networks, and a routing network selects top-k experts to activate for each token. Of DeepSeek's 748 billion total parameters, only about 80 billion are activated during reading and about 160 billion during generation, with the rest dormant—so floating-point operations per token are far lower than a dense model of equivalent total size. But the engineering challenge of MoE is that experts are distributed across GPUs, routing generates all-to-all communication, and load is inherently uneven—exactly the problem EPLB solves. Understanding MoE makes it clear that "wide expert parallelism" isn't simply chopping up the model, but matching memory capacity with activation sparsity, thereby shifting the bottleneck from compute to controllable communication and scheduling.

Move 2: EPLB Load Balancing—Like Opening Extra Checkout Counters at a Supermarket

Expert parallelism brings a practical problem: if many users are asking coding questions, the coding experts get hammered while the creative-writing experts sit idle. If one GPU takes 50 milliseconds while the other 127 finish in 5 milliseconds, the whole system still waits for the slowest one. DeepInfer's EPLB continuously monitors traffic and, upon detecting an overloaded expert, dynamically replicates that expert onto idle GPUs. It's like immediately opening four extra checkout counters when the supermarket line builds up—the cluster wastes almost no idle time.

Move 3: dSpark Speculative Decoding—Folding Bad Bets Early

dSpark isn't entirely new research, but baking it directly into the engine yields significant results. A small, fast auxiliary model drafts the next five words, and the large model verifies them in one pass. The problem with traditional speculative decoding is that even if the third word is wrong, the fourth and fifth have already been verified for nothing. DeepSeek adds a "confidence head" to the drafter that computes in real time the probability the large model will accept each word; upon finding the fourth word unreliable, it truncates early so the large model only verifies the first three. This "disciplined gambler" early fold improves cluster speed by another 4% to 7% without sacrificing accuracy.

Speculative Decoding is a technique that accelerates autoregressive generation by "drafting with a small model, verifying with a large one." When a large model generates tokens one by one, each step must move all activated weights, leaving much of the GPU's compute idle on memory bandwidth; speculative decoding lets a cheap draft model guess multiple candidate tokens at once, and the large model verifies them in parallel in a single forward pass. If accepted, multiple words are produced in one step, amortizing per-token bandwidth cost. Correctness relies on an accept-reject sampling rule where "the verification distribution matches the target distribution," so output quality is unchanged. The problem with traditional implementations is fixed draft length: once a token in the middle is rejected, all subsequent candidates are discarded. dSpark's confidence head essentially assigns an acceptance probability to each candidate, truncating low-confidence portions early to reduce wasted verification—which is why it can squeeze out another 4%–7% throughput without sacrificing accuracy.

Move 4: Bounded Replay—Coming Back Hours Later Without Rereading Everything

Long conversations are extremely storage-hungry for context, and AI companies are often forced to clear memory. The real bottleneck appears when a user leaves for two hours and comes back with a follow-up—servers typically have to reread the entire conversation from scratch. DeepSeek's bounded replay keeps active conversation memory in ultra-fast local RAM for only about three minutes, covering rapid back-and-forth chat; if RAM has been cleared, the server rereads only the last ~140 words. Whether the prompt is 1,000 words or 1 million, the time to fill in missing context is nearly negligible.

Rust Gateway and Hardware Roadmap: Skipping Hopper, Supporting Ascend

DeepSeek Recipe is written in Rust and handles tokenization before the prompt reaches the GPU. It splits prompts along structural boundaries, tokenizes in parallel across all CPU cores, and doubles as a security buffer: if a user tries to inject malicious system commands, the input is forcibly encoded as plain text, completely neutralizing the attack.

On hardware support, DeepInfer explicitly supports NVIDIA's new Blackwell architecture and Huawei's Ascend 950 chips, but skips the Hopper architecture's H100/H200. Amid tightening US export restrictions, this system processing 155 trillion tokens a day demonstrates the possibility of running at frontier scale on domestic silicon.

Hopper and Blackwell are two generations of NVIDIA data-center GPU architectures: Hopper (H100/H200) focuses on FP8 training and the Transformer Engine, while Blackwell further introduces larger memory, higher bandwidth, support for low-precision formats like FP4, and stronger multi-GPU interconnect. DeepSeek skipping Hopper to target Blackwell directly reflects both a natural preference for the latest generation in new cluster purchases and a sign that its kernels are specifically tuned for Blackwell's memory hierarchy and interconnect topology rather than running compatibly on older architectures. Huawei Ascend 950 belongs to the domestic AI accelerator lineage, with a software stack different from the CUDA ecosystem; DeepInfer's support indicates the engine abstracts at the operator level, enabling compilation and execution on non-NVIDIA backends. Amid tightening export controls, this multi-backend capability lets the same scheduling and parallelism strategy migrate across chips—the technical prerequisite for "running at frontier scale on domestic silicon" mentioned in the article.

What's Open-Sourced Isn't the Whole Factory: Three Barriers Remain

DeepInfer gives you the engine, not the whole factory. DeepSeek explicitly lists what it retains in the documentation: a 100PB storage cluster (pushing 10TB of data per second), a proprietary network routing system, and a global internet load balancer. In other words, they gave the world a high-performance commercial oven but kept the automated industrial bread line to themselves. To achieve the speeds described here, you still need an enterprise deployment of at least 128 GPUs running its flagship model, DeepSeek v4.1 Flash.

What This Means

The narrative of the past two years has been "whoever buys the most GPUs wins," and DeepSeek keeps proving that clever systems engineering can beat brute-force spending. This directly pressures cloud providers that charge high premiums for hosting open-source models, and developers of community inference engines like VLLM are clearly already digging through the code, preparing to bring these acceleration tricks into tools everyone can use. The focus of competition has shifted from "who builds the smartest model" to "who delivers intelligence at the lowest cost."

Key Takeaways

• DeepSeek published real data-center numbers: a single cluster processes 155 trillion tokens per day, a single GPU generates about $38 in revenue per hour, and gross margin is estimated at 80%—not cash-burning subsidies.
• It open-sourced the inference engine DeepInfer and the Rust gateway DeepSeek Recipe, with nearly 20,000 GitHub stars within 24 hours.
• Architecturally, it splits a 40-layer network into reading and writing halves, combined with a 748-billion-parameter, wake-on-demand Mixture-of-Experts design to cut compute waste at the foundation.
• Four engineering optimizations—wide expert parallelism, EPLB load balancing, dSpark speculative decoding, and bounded replay—together support low-cost, high-throughput serving.
• On hardware, it supports Blackwell and Huawei Ascend 950, skipping Hopper; but the 100PB storage, network routing, and global load balancing remain closed-source, and a full replication requires a 128-GPU-class enterprise deployment.

One-line Summary

DeepSeek open-sourced its inference engine DeepInfer, using systems engineering to push per-token cost to a level peers struggle to replicate.

Paragraph Summary

DeepSeek published real operational data from its data centers and open-sourced the inference engine DeepInfer and the Rust gateway DeepSeek Recipe, responding to doubts that its low prices rely on cash-burning subsidies. The article breaks down the sources of its cost advantage: a causal encoder-decoder architecture that splits a 40-layer network into reading and writing halves, and a Mixture-of-Experts design that wakes parameters on demand; at the engineering level, four moves—wide expert parallelism, EPLB dynamic load balancing, dSpark speculative decoding, and bounded replay—eliminate waste in memory traffic, queueing, and context rereading one by one. On hardware, it supports Blackwell and Huawei Ascend 950 while skipping Hopper. But the 100PB storage, network routing, and global load balancing remain closed-source, and a full replication requires a 128-GPU-class deployment. The focus of competition is shifting from model capability to the cost of delivering intelligence.