[KongchangAI]
ConceptPipeline Parallelism / PP

Pipeline Parallelism

A parallelism strategy that partitions a model by layer across different GPUs. Each token communicates across devices only once at each layer boundary, making it nearly immune to PCIe bandwidth constraints.

Timeline (last 90 days)

Oct 21

Pipeline parallelism splits the model into multiple stages by layer, places different stages on different devices, and executes them in a micro-batch pipeline, which relieves memory pressure but suffers from pipeline bubbles.

Unverified50%

All Facts (1)

Unverified

Pipeline parallelism splits the model into multiple stages by layer, places different stages on different devices, and executes them in a micro-batch pipeline, which relieves memory pressure but suffers from pipeline bubbles.

50%

Source Articles

TorchTitan Update: New SOTA Models, Performance at Scale, and TitanRLOct 21Does PCIe 4.0 x8/x8 Slow Down Dual-GPU Local LLM Inference?Oct 12