DeepSeek's Open-Source Infra Map: What Ordinary Programmers Should Understand

DeepSeek's open-source landscape goes far beyond model weights. From GPU operators to communication libraries, from distributed file systems to inference frameworks, it lays out layer by layer the entire infrastructure that keeps large models running. But with so many libraries and scattered documentation, it's hard for ordinary programmers to see the full picture. This article, based on a Bilibili creator's breakdown and cross-verified against multiple public sources, attempts to draw a complete map without diving into details: what problem each library solves, how they fit together, and which ideas you can take away directly.

The Big Picture: Think of a Large Model as a Factory

Before understanding these libraries, imagine a large model as a factory. At the bottom are operators—the most basic operations on the GPU; above that is communication and parallelism, enabling hundreds or thousands of GPUs to work together; above that is storage, responsible for feeding data and storing intermediate results; and at the top is the inference system, turning all of this into an external service.

Two timeline milestones are worth remembering. According to cross-referenced information from the Bilibili creator and GitHub project pages, DeepSeek^Ad once held a week-long open-source event, releasing one library per day; since then these libraries have been continuously updated and many new members added. The other milestone is October 20, when DeepInfer, the inference framework actually used in its online service, was open-sourced under a permissive Apache license.

To understand these libraries, you first need to know what the models they serve look like. DeepSeek uses a Mixture-of-Experts (MoE) model: a layer contains many experts, and each token is routed to only a few of them, so the total parameter count is huge while only a small portion participates in computation at any one time. This brings three new problems—each expert receives few tokens, making it hard to saturate the GPU; experts are distributed across multiple GPUs, and moving tokens back and forth makes communication a major cost; some experts are busy while others are idle, requiring load balancing across GPUs. Almost every library that follows addresses one of these three issues.

Mixture-of-Experts (MoE) is a sparsely activated model architecture. In DeepSeek's MoE, each Transformer layer contains dozens or even hundreds of "expert" feed-forward networks, but each token is assigned by a lightweight router to only a few of them (e.g., Top-2 or Top-6). Thus the total parameter count can be enormous while the parameters actually involved in a single forward pass are few, greatly reducing inference cost. The cost: experts are distributed across multiple GPUs, tokens need to be routed across GPUs, and communication volume surges; different experts naturally have uneven loads, easily causing some GPUs to idle while others are overloaded. Understanding this explains why libraries like DeepEP, EPLB, and DualPipe exist.

Layer 1: Operators—Making Every Operation Fast

Flash MLA addresses KV cache bloat. When a large model generates text, it must store the intermediate attention results for every previous token, which consumes a large amount of GPU memory. DeepSeek uses MLA to first compress this into a small vector before storing it, and Flash MLA is a GPU operator specifically optimized for efficiently reading this compressed format and computing attention. Another challenge is that request lengths vary; reserving memory based on the maximum length wastes GPU memory, so Flash MLA adopts paged management similar to an operating system's virtual memory—allocating only as much as needed.

DeepGEMM handles matrix multiplication, the most compute-intensive operation in large models. It uses low-precision FP8 to make the same hardware compute faster, with each block of values carrying its own scaling factor, so precision loss is minimal. The kernels are compiled just-in-time at runtime, exposing only a few core functions externally; the code is concise and suitable for reading and learning. For MoE, it also provides grouped matrix multiplication, grouping and computing together tokens sent to different experts; newer versions go further by fusing dispatch, two-layer computation, and combine into a single kernel called MegaMoE.

Several small, specialized libraries sit at the same layer: Tile Kernels writes operators in higher-level languages like TileLang; DeepSelect specializes in TopK selection; DeepJIT is a lightweight JIT compilation library. According to the creator and public information from Huawei, DeepSeek once collaborated with Huawei to bring TileLang to Ascend chips, resulting in Ascend versions of DeepGEMM and DeepEP.

MLA (Multi-head Latent Attention) is an attention variant proposed by DeepSeek. Its core idea is to first project and compress each token's Key/Value cache into a low-dimensional latent vector before storing it, then decompress and restore it during inference. Compared to traditional MHA/GQA, which must cache full K and V for each token, MLA's KV cache size can be reduced to a fraction of the original, thereby supporting longer contexts and larger batch sizes under the same GPU memory. Flash MLA is a GPU operator specifically optimized for this compressed format; it must handle details such as fusing decompression with attention computation and paged reads, which general-purpose attention kernels cannot do directly.

Layer 2: Communication and Parallelism—Keeping Thousands of GPUs Busy

MoE requires an all-to-all data exchange at every layer: sending tokens to the GPUs hosting the corresponding experts, then collecting them back after computation. DeepEP is a communication library written for this purpose, using NVLink within a machine and RDMA networks across machines, and supporting FP8 transmission to reduce data volume. It has two modes: training and prefill pursue high throughput, while decoding processes only one token at a time and pursues low latency. The latest second version merges both into a single interface and uses fewer compute units—less compute occupied by communication means more left for computation.

A communication library alone isn't enough; the GPU must not sit idle during communication. DualPipe is a pipeline parallelism schedule: it splits layers into segments placed on different GPUs, lets micro-batches from two directions enter simultaneously from both ends, and overlaps computation of one block with communication of another, greatly compressing "bubbles" (idle time).

For load balancing, EPLB's approach is to replicate popular experts several times and then distribute all replicas as evenly as possible across GPUs; prefill and decode use different strategies—the former is layered by machine, the latter is globally planned. There are also two early projects: LPLB uses linear programming to dynamically adjust load; Profile-Data publishes real training and inference performance profiling data, directly showing how computation and communication overlap.

NVLink and RDMA are two different tiers of interconnect technology. NVLink is a high-bandwidth direct bus between NVIDIA GPUs; multiple GPUs within a single server can exchange data via NVLink, with bandwidth far exceeding PCIe. RDMA (Remote Direct Memory Access) is used for cross-server communication, allowing one machine to directly read and write another machine's memory, bypassing the OS kernel with extremely low latency, commonly used in InfiniBand or RoCE networks. DeepEP is designed accordingly in layers: intra-machine via NVLink, inter-machine via RDMA, thereby squeezing bandwidth and reducing latency as much as possible in MoE's all-to-all communication.

Layer 3: Storage—Making the Entire Cluster Look Like One Disk

Training must read massive amounts of data, and inference must store and retrieve cached intermediate results; ordinary file systems can't keep up. 3FS is DeepSeek's distributed file system, built on SSDs and RDMA networks, making the entire cluster's disks behave like one high-speed large disk. It uses chain replication for writes, where data travels along a chain from head to tail, but reads can come from any replica on the chain, evenly spreading read pressure; files are split into small chunks distributed across multiple chains. 3FS serves four purposes: data preprocessing, reading data during training, saving checkpoints, and KV caching during inference. SmallPond, built on top of 3FS, is a lightweight data processing framework that lets people process large-scale data in familiar ways.

Top Layer: Inference System—Assembling Everything into a Service

The core of DeepSeek's publicly disclosed inference approach is two points: separating prefill and decode deployment, because the former is compute-bound and the latter is memory-bandwidth-bound; and using large-scale expert parallelism so each GPU hosts only a few experts, freeing up memory for larger batches. The official team also published a cost estimate: each H800 node processes about 73,000 input tokens or 14,800 output tokens per second; if all traffic were billed at the highest price, the theoretical profit margin would be about 545%—though the team emphasized this is only a theoretical value.

DeepInfer, open-sourced on October 20, was initially adapted from vLLM, integrates the aforementioned kernels, supports NVIDIA's latest-generation GPUs and Huawei Ascend, and has optimized scheduling. It has two interesting designs: speculative decoding, where a small draft model guesses five tokens at once and the large model verifies them in one pass, making correct guesses essentially free; and a deterministic mode that guarantees bitwise-identical output for the same input, facilitating debugging and reproduction.

We must honestly state the boundaries: this open-source release is not the entire production system. KV buffer management, cross-machine prefill-decode separation, and inter-replica load balancing—parts deeply coupled with internal infrastructure—were not released; currently only the V4.1 Flash model is supported, and Hopper and earlier GPUs are not supported.

Prefill and decode are the two stages of large model inference. In the prefill stage, the entire user prompt is processed at once, with all tokens computed in parallel—it is compute-intensive. In the decode stage, output is generated token by token, processing only one new token per step but repeatedly reading the huge KV cache and model weights—it is memory-bandwidth-intensive. The two have completely different resource bottlenecks, and mixing them in the same batch drags both down. Therefore DeepSeek adopts separated deployment: the prefill cluster focuses on compute, the decode cluster focuses on bandwidth, and together with large-scale expert parallelism, the number of experts per GPU is reduced to very few, freeing up memory for larger batches and improving overall throughput.

What Ordinary Programmers Can Take Away

Directly usable: kernel libraries can be installed separately and called from PyTorch; 3FS and SmallPond suit teams with large-scale data; DeepInfer suits deploying V4.1 Flash on supported new GPUs. However, many libraries require Hopper or Blackwell GPUs and won't work on ordinary GPUs.

More worth learning are the ideas: overlap computation and communication; manage variable-length requests with paging; use low precision with fine-grained scaling; replicate to spread hot spots; separate tasks with different characteristics into different deployments. These are useful when writing any high-concurrency system. One final piece of advice: don't dive straight into the code; first locate your problem on this map to see which layer it belongs to, then read the documentation for the corresponding library. To go deeper, DeepSeek's GitHub organization page is the best entry point.

One-line Summary

DeepSeek's open-source infrastructure spans four layers—operators, communication and parallelism, storage, and inference—with each library's problems and transferable ideas broken down one by one.

Paragraph Summary

Using the metaphor of "a large model as a factory," this article maps out DeepSeek's open-source infrastructure: at the bottom are GPU operators like Flash MLA and DeepGEMM, solving KV cache bloat and accelerating matrix multiplication; the middle layer comprises communication and parallelism libraries like DeepEP, DualPipe, and EPLB, addressing inter-expert communication, pipeline bubbles, and load imbalance in MoE models; the storage layer is the 3FS distributed file system and the SmallPond data processing framework; and the top layer is DeepInfer, the inference framework open-sourced on October 20, which integrates the aforementioned kernels and supports speculative decoding and deterministic mode. The article also notes that DeepInfer is not a complete production system and has limitations on GPUs and model versions, while distilling transferable engineering ideas such as computation-communication overlap, paged management, and low-precision scaling.