Context Parallelism
A distributed training strategy that partitions the input along the sequence-length dimension, used for parallel training over very long contexts or sequences.
Timeline (last 90 days)
Split-KV is an exact context-parallel (CP) method that maintains 85–91% weak scaling efficiency at a CP size of 64.
Split-KV achieves up to a 3.67x speedup over the fastest existing CP baseline.
Traditional context parallelism methods (such as Ring Attention) typically require transmitting the full KV or its chunks at every layer, with communication volume growing linearly with context length.
All Facts (3)
Split-KV is an exact context-parallel (CP) method that maintains 85–91% weak scaling efficiency at a CP size of 64.
50%UnverifiedSplit-KV achieves up to a 3.67x speedup over the fastest existing CP baseline.
50%UnverifiedTraditional context parallelism methods (such as Ring Attention) typically require transmitting the full KV or its chunks at every layer, with communication volume growing linearly with context length.
50%