Pipeline Parallelism

扩散模型的核心原理分为前向扩散过程（逐步添加高斯噪声）和反向去噪过程（从噪声中还原图像）两个阶段

Modern large model training typically employs a combination of multiple parallelism strategies simultaneously—including data parallelism, pipeline parallelism, tensor parallelism, and expert parallelism—forming a 3D or 4D parallel architecture.

块调度建议将张量并行（TP）限制在NVLink带宽最高的GPU子集内（同一个最小块），数据并行（DP）跨块分布，流水线并行（PP）沿拓扑层次展开

In distributed inference, pipeline parallelism (splitting by layer) has lower network bandwidth requirements but uneven device utilization; tensor parallelism (splitting the matrix operations of a single layer) communicates frequently and is latency-sensitive.

Megatron Core adopts a three-dimensional parallelism strategy combining tensor parallelism, pipeline parallelism, and data parallelism.

The solution splits the Transformer by layer, with the Mac handling layers 1–40 of each 256-token batch, then streaming the activations to the phone via USB-C, while the phone uses its own GPU to run layers 41–64, achieving pipeline parallelism.

The most common distribution strategy in microcontroller clusters is Pipeline Parallelism, which splits the Transformer model layer by layer

The trade-off of pipeline parallelism is that processing each token requires a full end-to-end wait, with inference latency accumulating linearly with the number of nodes

In topology-aware scheduling practice, GPUs in a tensor-parallel group should be preferentially placed within the same node, pipeline stages can span nodes but should ideally be on the same rack, and data-parallel replicas can be distributed flexibly.

The deployment uses a TP2 PP4 parallelism scheme: GPUs are paired in groups of two for tensor parallelism, forming four groups, with pipeline parallelism applied between groups.

Pipeline parallelism vertically partitions by model layers, with far less communication volume than tensor parallelism, but suffers from pipeline bubble issues that cause GPU idling.

Parallelization strategies for multi-GPU inference include tensor parallelism (splitting individual mathematical operations across multiple GPUs), pipeline parallelism (assigning different neural network layers to different GPUs), and data parallelism (replicating the entire model to process multiple batches of data).

This test most likely deploys multiple GPUs using pipeline parallelism, with the first half of the model's layers on the first card and the second half on the second card.

Tensor Parallelism (TP) splits individual Transformer layer weight matrices across different GPUs, requiring All-Reduce communication per layer forward pass; Pipeline Parallelism (PP) splits the model by layers with lower communication volume but has bubbles that reduce utilization