Unverified50% confidenceFactExact time
Tensor parallelism splits the weights of a single layer across multiple GPUs, is communication-intensive and suits high-bandwidth interconnects; pipeline parallelism splits layers across different devices, incurs bubble overhead and suits models with many layers.
1
Sources
50%
Confidence
Long-term
Relevance
11/4/2026
First Seen
Sources
Related Entities
Related Claims
Unverified流水线并行按模型的不同层将计算分配到不同设备，通信量仅为相邻层的激活张量，但存在流水线气泡问题79% similarUnverified张量并行将单层权重切分到多卡，延迟低但通信开销大；流水线并行按层切分，吞吐高但首token延迟长78% similarUnverifiedModel parallelism consists of two approaches: tensor parallelism, which splits tensors within a single layer across multiple cards, and pipeline parallelism, which assigns different layers to different devices.77% similarUnverifiedTensor Parallelism (TP) splits individual Transformer layer weight matrices across different GPUs, requiring All-Reduce communication per layer forward pass; Pipeline Parallelism (PP) splits the model by layers with lower communication volume but has bubbles that reduce utilization71% similarUnverifiedPipeline Parallelism vertically shards the model by layers, with different GPUs responsible for different layer groups; it is suited for cross-node deployment but requires micro-batch scheduling to reduce the idle waiting caused by pipeline bubbles.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1135842API
curl https://kongchang.com/api/v1/knowledge/claims/1135842MCP
get_claim(id=1135842)