CHARM+
An optimization mechanism for cross-hardware distributed attention computation. By decoupling multi-stream parallelism from inter- and intra-hardware communication, it reduces synchronization overhead caused by bandwidth imbalance and improves GPU utilization.
Timeline (last 90 days)
CHARM+ reorganizes originally sequential communication into multiple levels, achieving parallelism within the same level and overlapping computation with communication.
CHARM+'s Re-Merge Multistream mechanism uses multiple streams to split the originally serial "compute—communicate—compute" chain into interleavable segments, filling the waiting gaps of ring communication with useful computation.
CHARM+ decouples cross-hardware communication and intra-hardware communication, so that fast links do not have to keep pace with slow links, thereby eliminating synchronization waits caused by bandwidth imbalance.
The paper evaluates multiple hardware architectures on both forward and backward algorithms, all outperforming the current state-of-the-art method (Ring-Attention).
On a dual-NUMA PCIe architecture, CHARM+ achieves an average forward speedup of 1.3x and an average backward speedup of 1.7x, compared against Ring-Attention.
The speedup in backpropagation is generally higher than in the forward pass, because the backward computation involves more communication rounds and more complex dependencies, so the gains from communication optimization are more easily amplified on the backward path.
The ideas of splitting sequential communication into parallelizable levels and decoupling cross-hardware from intra-hardware communication also have reference value for other parallel strategies that rely on frequent collective communication.
As sequence length continues to grow, communication efficiency will become a more critical constraint than single-GPU compute.
All Facts (8)
CHARM+ reorganizes originally sequential communication into multiple levels, achieving parallelism within the same level and overlapping computation with communication.
50%UnverifiedCHARM+'s Re-Merge Multistream mechanism uses multiple streams to split the originally serial "compute—communicate—compute" chain into interleavable segments, filling the waiting gaps of ring communication with useful computation.
50%UnverifiedCHARM+ decouples cross-hardware communication and intra-hardware communication, so that fast links do not have to keep pace with slow links, thereby eliminating synchronization waits caused by bandwidth imbalance.
50%UnverifiedThe paper evaluates multiple hardware architectures on both forward and backward algorithms, all outperforming the current state-of-the-art method (Ring-Attention).
50%UnverifiedOn a dual-NUMA PCIe architecture, CHARM+ achieves an average forward speedup of 1.3x and an average backward speedup of 1.7x, compared against Ring-Attention.
50%UnverifiedThe speedup in backpropagation is generally higher than in the forward pass, because the backward computation involves more communication rounds and more complex dependencies, so the gains from communication optimization are more easily amplified on the backward path.
50%UnverifiedThe ideas of splitting sequential communication into parallelizable levels and decoupling cross-hardware from intra-hardware communication also have reference value for other parallel strategies that rely on frequent collective communication.
50%UnverifiedAs sequence length continues to grow, communication efficiency will become a more critical constraint than single-GPU compute.
50%