Sequence Parallelism
A distributed training strategy that partitions computation and activations along the sequence dimension, proposed by Megatron-LM. It is used in conjunction with tensor parallelism to reduce GPU memory pressure from activations during long-context training.
Timeline (last 90 days)
Sequence parallelism alleviates activation memory pressure in long-sequence scenarios by splitting computation along the sequence dimension
Sequence parallelism was formally proposed by Megatron-LM in a 2022 paper
Sequence parallelism splits the inputs of operators such as LayerNorm and Dropout along the sequence dimension, causing activation memory to decrease linearly with the degree of parallelism
All Facts (3)
Sequence parallelism alleviates activation memory pressure in long-sequence scenarios by splitting computation along the sequence dimension
50%UnverifiedSequence parallelism was formally proposed by Megatron-LM in a 2022 paper
50%UnverifiedSequence parallelism splits the inputs of operators such as LayerNorm and Dropout along the sequence dimension, causing activation memory to decrease linearly with the degree of parallelism
50%