Tensor Parallelism

扩散模型的核心原理分为前向扩散过程（逐步添加高斯噪声）和反向去噪过程（从噪声中还原图像）两个阶段

Modern large model training typically employs a combination of multiple parallelism strategies simultaneously—including data parallelism, pipeline parallelism, tensor parallelism, and expert parallelism—forming a 3D or 4D parallel architecture.

块调度建议将张量并行（TP）限制在NVLink带宽最高的GPU子集内（同一个最小块），数据并行（DP）跨块分布，流水线并行（PP）沿拓扑层次展开

Each token generated by a large model requires a large number of matrix multiplication operations on the GPU, which is the core computational step of the attention mechanism in the Transformer architecture.

In distributed inference, pipeline parallelism (splitting by layer) has lower network bandwidth requirements but uneven device utilization; tensor parallelism (splitting the matrix operations of a single layer) communicates frequently and is latency-sensitive.

Tensor parallelism splits the weight matrix of the same layer and places it across multiple machines, requiring one All-Reduce communication per layer computed, is extremely sensitive to bandwidth and latency, and is usually only practical under high-speed interconnects such as NVLink.

张量并行的通信瓶颈不是单次传输的数据量，而是同步次数乘以每次同步的固有延迟。

在张量并行模式下，模型权重在推理开始前已分片常驻各 GPU 显存，跨 PCIe 传输的只有前向传播中的中间激活张量，而非权重。

Tensor parallelism splits a single matrix operation across multiple GPUs working together, requiring an AllReduce communication to aggregate results after each operation.

Megatron Core adopts a three-dimensional parallelism strategy combining tensor parallelism, pipeline parallelism, and data parallelism.

Tensor parallelism is typically used only between NVLink-connected GPUs within the same node; cross-node deployment severely slows training throughput due to insufficient InfiniBand bandwidth

Tensor parallelism splits the matrix operations within a single layer across multiple GPUs for parallel computation

The column/row splitting scheme for tensor parallelism was first proposed by Megatron-LM

In Megatron-LM's tensor parallelism scheme, the Query, Key, and Value projection matrices are split by column, the output projection matrix is split by row, and an All-Reduce communication is inserted at each end

大语言模型多卡张量并行时，每一层前向传播都需要在卡间同步激活值，带宽瓶颈会直接拉低整体推理吞吐

In tensor-parallel inference, AllReduce operations occur at every layer, and for large models with tens or even hundreds of layers, the cumulative volume of communicated data is considerable.

ERRC (Entropy Reinvestment Residual Correction) is a solution targeting GPU inter-communication bottlenecks in tensor-parallel LLM inference; it reduces communication volume by compressing exchanged data and uses the savings to compensate for quantization errors.

Tensor parallelism requires extremely low-latency all-reduce operations and has the strongest dependency on NVLink bandwidth.

In topology-aware scheduling practice, GPUs in a tensor-parallel group should be preferentially placed within the same node, pipeline stages can span nodes but should ideally be on the same rack, and data-parallel replicas can be distributed flexibly.

The deployment uses a TP2 PP4 parallelism scheme: GPUs are paired in groups of two for tensor parallelism, forming four groups, with pipeline parallelism applied between groups.