Split-KV
An exact context parallelism (CP) method that keeps key/value shards local to each device. In each layer, attention communicates only tensors comparable in size to the queries, decoupling communication volume from context length and making it suitable for asymmetric training scenarios with very short queries and very long contexts.
Timeline (last 90 days)
Split-KV is an exact context-parallel (CP) method that maintains 85–91% weak scaling efficiency at a CP size of 64.
Split-KV achieves up to a 3.67x speedup over the fastest existing CP baseline.
The core mechanism of Split-KV is to keep key/value shards local to each device, so that each layer's attention only communicates tensors comparable in size to the query, thereby decoupling communication volume from context length.
Split-KV uses a cost-aware longest-processing-time-first (LPT) strategy to allocate context KV blocks, with a load imbalance rate below 1%.
Split-KV's LPT scheduling planning is three orders of magnitude faster than the best load-balancing baseline.
Split-KV's code is open-sourced in the verl repository on GitHub at https://github.com/zxgx/verl.
Split-KV is an exact method; its computation results are strictly consistent with the non-sharded case and it does not sacrifice numerical correctness.
All Facts (7)
Split-KV uses a cost-aware longest-processing-time-first (LPT) strategy to allocate context KV blocks, with a load imbalance rate below 1%.
50%UnverifiedSplit-KV achieves up to a 3.67x speedup over the fastest existing CP baseline.
50%UnverifiedThe core mechanism of Split-KV is to keep key/value shards local to each device, so that each layer's attention only communicates tensors comparable in size to the query, thereby decoupling communication volume from context length.
50%UnverifiedSplit-KV is an exact context-parallel (CP) method that maintains 85–91% weak scaling efficiency at a CP size of 64.
50%UnverifiedSplit-KV's LPT scheduling planning is three orders of magnitude faster than the best load-balancing baseline.
50%UnverifiedSplit-KV is an exact method; its computation results are strictly consistent with the non-sharded case and it does not sacrifice numerical correctness.
50%UnverifiedSplit-KV's code is open-sourced in the verl repository on GitHub at https://github.com/zxgx/verl.
50%