GB200 NVL72 Block Scheduling in Practice: How Slurm Squeezes Every Bit of Rack-Level NVLink Performance

Introduction: GPU Cluster Architecture Is Undergoing a Paradigm Shift

NVIDIA GB200 NVL72 is not a simple hardware upgrade—it's a fundamental transformation in how GPU clusters are built. It extends NVLink coherency from within a single node to an entire rack—72 GPUs are fully interconnected through NVLink Switches, forming a unified high-bandwidth communication domain.

But no matter how powerful the hardware is, it's wasted if scheduling can't keep up. A critical question faces every operations and architecture team: How do you maximize efficiency on such a highly interconnected system?

The answer points to Slurm's Block Scheduling. This article breaks down the architectural characteristics of GB200 NVL72, explains why block scheduling is the key to unlocking its full potential, and provides actionable configuration recommendations.

GB200 NVL72: What Rack-Level NVLink Coherency Brings to the Table

The 72-GPU Fully Interconnected Architecture

The core innovation of GB200 NVL72 lies in extending NVLink interconnect from within a node to an entire rack. A single NVL72 rack contains 72 GPUs fully interconnected through NVLink Switches, forming a massive coherent memory domain.

What does this mean? Any two GPUs within the rack can communicate directly via NVLink, with bandwidth far exceeding InfiniBand or Ethernet. For large-scale model training, the long-standing challenge of cross-node communication is dramatically mitigated within the rack—72 GPUs are essentially one "super node."

Traditional Node Boundaries Are Broken

In previous DGX clusters, 8 GPUs formed a node, with intra-node communication over NVLink and inter-node communication over InfiniBand. The scheduler only needed to allocate by node—simple logic.

But NVL72 completely breaks this boundary. 72 GPUs share the same NVLink domain, the concept of a node is weakened, and what replaces it is a more complex topological hierarchy. This places entirely new demands on the scheduling system.

Why Traditional Scheduling Fails on NVL72

Traditional Slurm schedulers allocate resources in node-sized units, which leads to two serious problems on NVL72 architecture:

Resource fragmentation intensifies. Suppose several jobs of different sizes are submitted sequentially to a rack. After default allocation, idle GPUs may be scattered throughout the rack. Subsequent large jobs may have to wait in the queue even though the total number of free GPUs is sufficient, because no contiguous block can be found. This is the classic "Swiss cheese" problem—holes everywhere, but none big enough.

Communication efficiency takes a hit. The NVLink Switch topology is not perfectly flat. Although all 72 GPUs are interconnected, communication paths through different switch levels have varying latency and effective bandwidth. If 16 GPUs are allocated to topologically scattered positions, collective communication operations like AllReduce will perform noticeably below theoretical peak.

In short, traditional scheduling on NVL72 wastes both resources and bandwidth.

Slurm Block Scheduling: Core Mechanism and Key Advantages

The Basic Principle of Block Scheduling

The core idea of Block Scheduling is: partition the 72 GPUs into multiple logical "blocks" according to the NVLink topology, and allocate jobs in block-sized units.

Specifically, based on the NVLink Switch connection hierarchy, the 72 GPUs can be divided into blocks of various granularities—for example, 9 blocks of 8 GPUs, 4 blocks of 18 GPUs, or even 2 blocks of 36 GPUs. GPUs within each block are physically adjacent in the topology and share the shortest NVLink communication paths.

When a job requests 16 GPUs, the block scheduler doesn't arbitrarily pick 16 free GPUs. Instead, it allocates two adjacent 8-GPU blocks, ensuring those 16 GPUs form a topologically compact subset.

Three Key Advantages

Advantage 1: Significantly Reduced Communication Latency

Topology-aware allocation reduces the number of hops data must traverse in the NVLink network. For collective communication operations like AllReduce and AllGather, fewer hops directly translate to lower latency and higher effective bandwidth. In large model training, communication overhead often accounts for more than 30% of total training time, making optimizations in this area highly impactful.

Advantage 2: Dramatically Reduced Resource Fragmentation

Block scheduling allocates and reclaims resources in a structured manner. When a job completes, it releases complete blocks rather than scattered GPUs. Subsequent jobs can directly reuse these complete blocks, avoiding the awkward situation of "having free GPUs but unable to assemble a contiguous block."

Advantage 3: Overall System Throughput Improvement

Reduced fragmentation means higher GPU utilization and shorter queue wait times. Under high cluster load scenarios (which is the norm in production environments), the throughput gains from block scheduling are especially pronounced. Multiple benchmarks show that compared to default scheduling policies, block scheduling can improve overall system utilization by 15%-25%.

Implementation: Slurm Configuration and Workload Optimization

Key Slurm-Side Configuration

Enabling block scheduling in Slurm for GB200 NVL72 requires attention to the following configuration aspects:

Topology Plugin Configuration

Enable Slurm's topology plugin and inject the NVL72's NVLink topology information into the scheduler. This is the foundation of block scheduling—the scheduler must know which GPUs have shorter, higher-bandwidth communication paths to make topology-aware allocation decisions.

Partition and Block Granularity Definition

Define reasonable block sizes based on the NVL72's physical topology hierarchy. Common partition schemes include:

• 8-GPU blocks: suitable for medium-scale tensor parallel jobs
• 18-GPU blocks: suitable for training tasks requiring greater parallelism
• 36-GPU blocks: suitable for half-rack-scale large jobs
• 72-GPU blocks: exclusive use of the entire rack, suitable for ultra-large-scale training

Define these block sizes as schedulable resource units in Slurm's partition configuration.

Job Constraints and Alignment Policies

Use Slurm's constraint mechanism to guide users to align their requested GPU counts with predefined block sizes. For example, a job requesting 12 GPUs can be automatically rounded up to 16 (two 8-GPU blocks), or the user can be prompted to adjust their request. While this alignment may introduce minor resource redundancy, the scheduling efficiency gains far outweigh the cost.

Workload-Side Optimization Recommendations

With hardware and scheduling properly configured, workloads themselves also need adjustment to maximize results:

Align Parallelism Strategies with Block Hierarchy

This is the most important point. Constrain tensor parallelism (TP) to the GPU subset with the highest NVLink bandwidth (typically within the same smallest block), distribute data parallelism (DP) across blocks, and unfold pipeline parallelism (PP) along the topology hierarchy. This way, each parallelism strategy matches the most appropriate communication bandwidth tier.

Choose Job Sizes That Are Integer Multiples of Block Sizes

Requested GPU counts should ideally be integer multiples of block sizes like 8, 18, or 36. For example, if you need 32 GPUs, adjusting to 36 (two 18-GPU blocks) is typically more efficient than forcing 32—the communication optimization benefits from the extra 4 GPUs often exceed the cost of resource redundancy.

Fully Leverage NVLink Coherent Memory

At the programming level, utilize the coherent memory access capabilities provided by NVLink to reduce explicit inter-GPU data transfers. Communication libraries like NCCL have been deeply optimized for NVLink topologies—ensure you're using the latest version with topology information correctly configured.

Performance Impact and Future Directions

Block scheduling improves GB200 NVL72 efficiency across multiple dimensions:

• Per-job dimension: Topology-aware GPU allocation reduces collective communication overhead, significantly improving training iteration speed
• System dimension: Reduced fragmentation leads to higher GPU utilization and shorter queue wait times
• Operations dimension: Structured resource management simplifies fault isolation and capacity planning

Cross-Rack Challenges

As AI model parameter counts continue to grow, single training jobs spanning multiple NVL72 racks will become the norm. At that point, the block scheduling philosophy needs to scale upward: not only optimizing GPU allocation within a rack, but also optimizing network topology across racks—selecting rack combinations with optimal InfiniBand connectivity to minimize cross-rack communication performance penalties.

This is essentially a multi-level topology-aware scheduling problem: NVLink block scheduling within racks, network topology-aware scheduling across racks, with both working in concert to maintain efficiency at ultra-large scale.

Co-Evolution of Scheduling and Hardware

The combination of GB200 NVL72 and Slurm block scheduling reveals an increasingly clear trend: hardware architecture innovation must co-evolve with scheduling software to truly deliver on performance promises.

No matter how strong the interconnect bandwidth, if the scheduler doesn't understand the topology, the GPU combinations it allocates still won't perform well. Conversely, no matter how sophisticated the scheduling algorithm, if the underlying hardware doesn't provide sufficient topology information and flexibility, there's nothing to work with.

For teams planning or already deploying GB200 NVL72, block scheduling is not an optional optimization—it's a necessary condition for fully realizing the return on hardware investment.