A Complete Guide to Distributed LLM Training: DP/TP/PP/SP Explained

When a large model won't fit on one GPU, combine DP, TP, PP, and SP to scale capacity and throughput.
Starting from the token-by-token forward computation of large models, the article notes that model parameters routinely reach hundreds of GB while a single GPU has only tens of GB—a fundamental conflict that drives distributed parallelism. It then breaks down four strategies: DP replicates the model and scales throughput but doesn't reduce per-card memory; TP splits weights within layers with frequent communication, suited to high-bandwidth intra-node interconnects; PP splits by layer with low communication and cross-node scalability but suffers from bubbles that micro-batches must fill; SP splits activations along the sequence dimension and often pairs with TP for long contexts. It concludes with practical decision logic—first check if the model fits on one card, then decide whether to scale throughput or use model parallelism—and clarifies two common misconceptions: quantization doesn't equal single-card deployability, and DP can't scale capacity.
Why Can't a Single GPU Run Large Models?
A large model is essentially a Decoder-Only neural network that generates output token by token, requiring a forward pass for every word it produces. This computation can be simplified as: input data X multiplied by model parameters W yields result Y. For the computation to be fast enough, both X and W need to be loaded into GPU memory.
The problem is that model size far exceeds the memory capacity of a single GPU. According to a demonstration in a Bilibili tutorial, the parameter files of today's mainstream large models routinely reach hundreds of GB, while even newer consumer or data-center GPUs often have only tens of GB of memory per card. The presenter used the old GPU in his own desktop as an example—it has only around a dozen GB of memory and simply cannot load a model of hundreds of GB directly.

Even on GPU rental platforms like AutoDL, where available models such as H800, 5090, A800, and V100 offer anywhere from tens of GB to 80GB per card, a full-scale large model still won't fit. Quantization can reduce the size, but the quantized versions shown in the tutorial still reach over a hundred GB, which a single card still cannot hold.

This is the fundamental driving force behind distributed parallel training and deployment: when a model doesn't fit on one card, the computation and storage must be split across multiple cards working together.
Data Parallelism (DP): The Most Intuitive Way to Scale
Data parallelism solves the problem of "too much computation, too slow on a single card." The idea is to replicate the same model parameters onto multiple GPUs, have each card process a different data batch, compute gradients independently, and then synchronize the gradients through communication.
The advantage of this approach is that it's simple to implement and non-invasive to the model architecture—almost every training framework supports it natively. The cost is that each card must hold a complete copy of the model parameters, so memory usage isn't reduced—it scales throughput, not model capacity. When the model itself is too large for a single card, DP can't help.
Model Parallelism (TP/PP): Splitting the Model Apart
Model parallelism targets exactly the "model doesn't fit" scenario by splitting the model itself across multiple cards. The tutorial focuses on dissecting two mainstream approaches.
Tensor Parallelism (TP)
Tensor parallelism splits within a layer. Taking matrix multiplication as an example, the weight matrix can be split by row or by column across different GPUs, with each card computing only a portion and then combining the results into a complete output through communication operations like All-Reduce. Its granularity is fine and communication is frequent, placing high demands on inter-GPU bandwidth, so it's typically confined to within a single machine or a high-bandwidth interconnect node.
All-Reduce is one of the most commonly used collective communication primitives in distributed computing: all participating devices each hold a piece of data, and after communication completes, every device obtains the reduction result (such as a sum) of all devices' data. In tensor parallelism, after the column-split weight matrices each compute their partial results, All-Reduce is needed to sum the partial results across cards to obtain the complete output tensor. Related operations include All-Gather (concatenating each card's shard into a complete tensor) and Reduce-Scatter (reduce first, then scatter the shards). The time these operations take is mainly determined by inter-GPU bandwidth, so TP is typically used only within a single machine with high-speed interconnects like NVLink; once you go across nodes over ordinary networks, communication overhead quickly eats up the gains from parallelism.
Pipeline Parallelism (PP)
Pipeline parallelism splits by layer. For example, the first several layers go on GPU 0, the middle layers on GPU 1, and the later layers on GPU 2, with data flowing through the cards in sequence like an assembly line. Its communication volume is smaller than TP, making it suitable for cross-node scaling, but it introduces the "bubble" problem—some cards sit idle waiting for upstream results, which must be filled by micro-batch scheduling.

"Bubbles" refer to the idle waiting time of devices in a pipeline. If you split the model into several segments by layer and place them on different GPUs, processing only one batch at a time, then only one card is working at any given moment while the rest wait for upstream output—device utilization is extremely low. The solution is to split a large batch into multiple micro-batches, letting each segment process different micro-batches continuously like a factory assembly line, thereby filling the idle time. The bubble ratio is roughly proportional to "pipeline stages minus one" and inversely proportional to the number of micro-batches, so increasing micro-batches or reducing the number of splits both improve utilization—but the former is limited by memory and the latter by per-card capacity, requiring trade-offs in practice.
Sequence Parallelism (SP): A Complement for Long Contexts
Sequence parallelism splits along the sequence dimension, dividing a long sequence into multiple segments for different devices to process. It's commonly used in long-context scenarios and paired with tensor parallelism to further reduce per-card memory pressure from activations. The tutorial positions it as a complement to the DP/TP/PP system rather than a replacement.
Practical Perspective: First Identify Where the Bottleneck Is
The practical logic the tutorial lays out is clear: first determine whether the model fits on a single card. If it fits but runs slowly, use data parallelism to boost throughput; if it doesn't fit, go to model parallelism—TP within layers, PP across layers, and add SP for long sequences. Real deployments often use hybrid parallelism—TP kept within a small scope to ensure communication efficiency, PP scaling across nodes, and DP at the outermost layer to amplify throughput.
According to the Bilibili tutorial, this parallelism system isn't deeply coupled with the model's own neural network architecture and is understandable even to beginners—it's a very practical need for both private deployment and model training.

Common Misconceptions
One easily overlooked point: quantization does not mean a model can be deployed on a single card. The quantized models demonstrated in the tutorial are still enormous—quantization only reduces the number of bits per parameter and cannot change the basic fact that "model capacity exceeds single-card memory." Another misconception is treating DP as a universal scaling method—it only scales throughput, not capacity, and can't help when the model doesn't fit.
Only by understanding the splitting dimensions and communication costs of DP, TP, PP, and SP can you combine them into a reasonable parallel scheme for real-world scenarios.
Quantization refers to storing and computing parameters at lower numerical precision, commonly by compressing 16-bit floating-point weights down to 8-bit, 4-bit, or even lower, so that model size shrinks proportionally. It falls into two categories: weight-only quantization and quantization of both weights and activations; the former is simpler to deploy with less accuracy loss and is the most common form in private deployment. But quantization has a clear floor: 4-bit quantization can only compress size to roughly a quarter of the original, so a model of hundreds of GB still ends up at tens of GB or more after quantization—plus KV Cache and activation overhead during inference, a single card may still not fit it. Quantization should therefore be viewed as an optimization for reducing deployment costs, not as a capacity solution that replaces parallelism.
Background Notes
Activations are the intermediate tensors produced during the forward pass, which must be retained for gradient computation during backpropagation and therefore also consume memory. Model parameters are static and their size is predictable, whereas activations grow with batch size and sequence length—in long-context scenarios they may even exceed the parameters themselves and become the memory bottleneck. Sequence parallelism targets exactly this overhead: it distributes the activations split along the sequence dimension across devices and aggregates them via communication where the full context is needed (such as LayerNorm and some operations in attention). Because it splits activations rather than weights, it's often stacked with tensor parallelism to form a "weight splitting + activation splitting" combination that further lowers per-card memory pressure.
Related articles

Dify Workflow in Practice: A Guide to Local Deployment and Private Knowledge Base Agents
Based on a Bilibili Dify course: Windows Docker local deployment, MySQL connection setup, five app types and workflow node cases, external LLM integration choices, and the complete path to publishing apps as sites or APIs.

DeepSeek Harness Step-by-Step Guide: From Installation and Setup to Plugin Practice
DeepSeek Harness step-by-step guide: Node.js setup, client installation, and three plugin installation methods to get you started fast with this open-source AI Agent framework.

MCP Beginner's Guide: The Standardized Protocol for LLMs to Connect to External Tools
Based on open course content from Bilibili, this article explains the core definition and value of the MCP protocol: why large models need a standardized tool connection protocol, how MCP can encapsulate and reuse traditional software interfaces, and the conceptual distinction between MCP and Skill, offering beginner guidance for Agent developers.