CHARM+: How Cross-Hardware Attention Breaks Through Distributed Communication Bottlenecks

Long-sequence inference has become routine in large model training, and distributed attention schemes such as Ring-Attention are widely adopted. However, communication dependencies make it difficult to fully overlap computation and communication, limiting GPU utilization. In cross-NUMA or cross-node architectures, bandwidth imbalance further turns frequent synchronization into a drag on overall performance. CHARM+ (Cross-Hardware Attention with Re-Merge Multistream Mechanism) is an optimization scheme proposed precisely to address this pain point.

Where Does the Communication Bottleneck of Distributed Attention Come From?

Ring-Attention splits sequences across multiple GPUs and progressively aggregates attention results through ring communication. This pattern inherently depends on sequential communication: each round of computation must wait for the previous round of communication to complete, compressing the room for overlapping computation and communication.

The problem is further amplified in cross-hardware scenarios. In NUMA architectures or cross-node deployments, bandwidth across different links is not symmetric—local links are fast while cross-hardware links are slow. Frequent synchronization operations cause fast links to idle while waiting for slow links, and overall throughput is held back by the slowest link. The paper points out that the synchronization overhead caused by this bandwidth imbalance is an obstacle that cross-hardware distributed attention cannot easily avoid.

Ring-Attention is a distributed attention implementation that splits long sequences across multiple GPUs and progressively aggregates attention results through a ring topology. Its core idea is to partition the sequence into blocks distributed across devices, with each GPU holding a portion of the queries (Q), while blocks of keys (K) and values (V) are passed sequentially along the ring; each round computes local attention and accumulates partial results until all KV blocks have been traversed. This design decouples per-GPU memory usage from sequence length, and is therefore widely used in long-context training and inference. But its cost is that communication and computation strictly alternate: each round must wait for the previous round's KV block to arrive before computation can begin, forming a natural serial dependency chain with a narrow overlap window between computation and communication—especially pronounced when bandwidth is limited or links are imbalanced.

NUMA (Non-Uniform Memory Access) is a multiprocessor architecture: each CPU or GPU has local memory with low access latency and high bandwidth, while accessing remote memory of other nodes requires going through interconnect links with higher latency and lower bandwidth. As a result, link bandwidth between different devices within the same machine is not symmetric, forming a "mixed fast-and-slow" topology. This problem is even more prominent in cross-node deployments, where the bandwidth gap between intra-node (e.g., NVLink) and inter-node (e.g., InfiniBand or Ethernet) links can reach several times. In workloads requiring frequent synchronization such as distributed attention, fast links must wait for slow links to complete, and overall throughput is held back by the slowest segment—this is precisely the root of the bandwidth-imbalance synchronization overhead emphasized in the article.

Two Core Designs of CHARM+

Communication Tiering and Multistream Parallelism

CHARM+ reorganizes originally sequential communication into multiple tiers, achieving parallelism within the same tier and overlapping computation with communication. In this way, the serial chain in ring communication—"wait for the previous step to finish before proceeding to the next"—is broken apart, and GPUs still have local work to do while waiting for remote data.

Multistream parallelism refers to using multiple CUDA streams on the GPU to schedule computation kernels and communication operations simultaneously. Tasks in different streams can execute concurrently, and together with asynchronous copy engines, computation kernels can continue processing local blocks that are already ready while waiting for remote data to arrive. CHARM+'s Re-Merge Multistream mechanism leverages multistreaming to break the originally serial "compute—communicate—compute" chain into interleavable segments, thereby filling effective computation into the waiting gaps of ring communication and improving GPU utilization. The effectiveness of multistream parallelism depends on task partitioning granularity and dependency management: too coarse a granularity makes overlap difficult, while too fine a granularity increases scheduling overhead.

Decoupling Cross-Hardware and Intra-Hardware Communication

The second design is to handle cross-hardware communication and intra-hardware communication separately. Cross-hardware links have low bandwidth and high latency, while intra-hardware links are much faster. After decoupling the two, fast links no longer need to synchronize to the rhythm of slow links, and synchronization waits caused by bandwidth imbalance are eliminated.

These two designs together constitute the Re-Merge Multistream mechanism: multistream parallelism is responsible for exploiting overlap opportunities, while tiering and decoupling are responsible for eliminating unnecessary synchronization points.

Measured Speedups: Dual NUMA vs. Dual Node Comparison

The paper evaluates multiple hardware architectures on both forward and backward algorithms, all outperforming current state-of-the-art methods. The specific data are as follows:

• Dual NUMA PCIe architecture: 1.3× average forward speedup and 1.7× average backward speedup, compared against Ring-Attention.
• Dual-node NVLink architecture: 2.8× average forward speedup and 2.9× average backward speedup.

You may not have noticed that the speedup for backpropagation is generally higher than for the forward pass. Considering that backward computation involves more communication rounds and more complex dependencies, the benefits of communication optimization are more easily amplified on the backward path—a result that aligns with intuition.

The speedup on NVLink architectures is significantly higher than on PCIe architectures. Although cross-node NVLink has higher bandwidth than PCIe, the bandwidth gap between inter-node and intra-node links still exists, and CHARM+'s decoupling design yields greater benefits in this "mixed fast-and-slow" topology.

Practical Significance for Long-Sequence Inference

From an engineering perspective, the value of CHARM+ lies not only in speedup numbers. It targets the typical deployment forms of distributed attention in real clusters—NUMA nodes, multi-machine multi-GPU setups, and uneven link bandwidth. In these scenarios, the gap between theoretical compute power and actual utilization often comes from communication synchronization rather than computation itself.

Splitting sequential communication into parallelizable tiers and decoupling cross-hardware from intra-hardware communication—this approach also has reference value for other parallel strategies that rely on frequent collective communication. As sequence lengths continue to grow, communication efficiency will become a more critical constraint than per-GPU compute power, and work such as CHARM+ on cross-hardware communication optimization is pushing outward the scalability boundary of attention mechanisms.

One-line Summary

CHARM+ eliminates the synchronization bottleneck of distributed attention through communication tiering and multistream parallelism, as well as decoupling cross-hardware from intra-hardware communication, achieving up to 2.9× speedup on dual-NUMA and dual-node architectures.

Paragraph Summary

Ring-Attention, widely used in long-sequence inference, suffers from sequential communication dependencies that make it difficult to overlap computation and communication; in cross-NUMA or cross-node deployments, bandwidth imbalance further makes frequent synchronization a performance drag. CHARM+ proposes two core designs: first, reorganizing sequential communication into multiple tiers to achieve intra-tier parallelism and computation-communication overlap; second, decoupling cross-hardware from intra-hardware communication so that fast links no longer synchronize with slow links. Together they form the Re-Merge Multistream mechanism. Measurements show 1.3× forward and 1.7× backward speedup on dual-NUMA PCIe architectures, and 2.8× forward and 2.9× backward speedup on dual-node NVLink architectures, with backward speedups generally higher than forward and NVLink gains greater than PCIe. This work targets the typical deployment forms in real clusters where link bandwidth is uneven, and also has reference value for other parallel strategies that rely on frequent collective communication.