NUMA
A multiprocessor memory architecture in which each processor has local memory with low latency and high bandwidth, while remote access goes through interconnection links with higher latency and lower bandwidth, forming a hybrid topology of fast and slow access.
Timeline (last 90 days)
On a dual-NUMA PCIe architecture, CHARM+ achieves an average forward speedup of 1.3x and an average backward speedup of 1.7x, compared against Ring-Attention.
In NUMA architectures, accessing local memory has low latency and high bandwidth, while accessing remote memory has higher latency and lower bandwidth, causing unequal link bandwidth between different devices within the same machine.
AI cluster day-one readiness requires the Kubernetes side to rely on the NVIDIA Device Plugin to register GPUs as schedulable resources, and the scheduler and topology manager must correctly recognize NUMA and GPU affinity.
In heterogeneous backends, PCIe bandwidth is typically far lower than NVLink and will become the bottleneck for cross-device scheduling; placing the draft model on a device under the same PCIe root complex as the target model, or directly on the CPU, is often more worthwhile than moving data across NUMA nodes.
For inference solutions that rely on host memory offload, the NUMA node where the expert data offloaded to host memory resides should be kept on the same node as the GPU accessing it, otherwise the performance loss will be mistakenly attributed to the offload mechanism itself.
On a rented machine, inference speed was 1.17 to 1.51 times slower when the two GPUs were attached to different NUMA nodes.
Under a NUMA architecture, each CPU accesses its local memory far faster than remote memory under other CPUs' jurisdiction, and cross-NUMA-node data transfers significantly reduce actual bandwidth utilization.
All Facts (7)
On a dual-NUMA PCIe architecture, CHARM+ achieves an average forward speedup of 1.3x and an average backward speedup of 1.7x, compared against Ring-Attention.
50%UnverifiedIn NUMA architectures, accessing local memory has low latency and high bandwidth, while accessing remote memory has higher latency and lower bandwidth, causing unequal link bandwidth between different devices within the same machine.
50%UnverifiedAI cluster day-one readiness requires the Kubernetes side to rely on the NVIDIA Device Plugin to register GPUs as schedulable resources, and the scheduler and topology manager must correctly recognize NUMA and GPU affinity.
50%UnverifiedIn heterogeneous backends, PCIe bandwidth is typically far lower than NVLink and will become the bottleneck for cross-device scheduling; placing the draft model on a device under the same PCIe root complex as the target model, or directly on the CPU, is often more worthwhile than moving data across NUMA nodes.
50%UnverifiedFor inference solutions that rely on host memory offload, the NUMA node where the expert data offloaded to host memory resides should be kept on the same node as the GPU accessing it, otherwise the performance loss will be mistakenly attributed to the offload mechanism itself.
50%UnverifiedOn a rented machine, inference speed was 1.17 to 1.51 times slower when the two GPUs were attached to different NUMA nodes.
50%UnverifiedUnder a NUMA architecture, each CPU accesses its local memory far faster than remote memory under other CPUs' jurisdiction, and cross-NUMA-node data transfers significantly reduce actual bandwidth utilization.
50%