NVIDIA Topograph: Making GPU Workload Scheduling Network Topology-Aware

AI factories are fundamentally power-constrained systems that deliver maximum value only when fully optimized. Within this optimization framework, GPU workload placement is often an overlooked yet critically important factor. When hundreds or thousands of GPUs are interconnected through complex, multi-tier networks, which nodes and GPUs a task gets scheduled onto directly determines communication overhead — and in turn, the throughput efficiency of the entire cluster.

Why Workload Placement Matters

In large-scale distributed training and inference scenarios, GPUs need to frequently exchange gradients, activations, and intermediate results. This communication is not uniformly distributed — GPUs within the same server are connected via NVLink's high-speed interconnect, while cross-rack and cross-switch communication must traverse network links with lower bandwidth and higher latency.

If a scheduler has no awareness of the underlying hardware topology, it may scatter communication-intensive tasks across nodes that are far apart on the network, forcing large volumes of data to travel back and forth over low-bandwidth links. This kind of "poor placement" turns communication into a bottleneck: even if every GPU's compute capacity is fully utilized, overall cluster utilization remains low. For AI factories where power consumption is a hard constraint, this means the effective computation delivered per watt steadily shrinks.

NVLink is NVIDIA's high-speed GPU interconnect bus for direct communication between multiple GPUs within the same server. Fourth-generation NVLink delivers unidirectional bandwidth of up to 900 GB/s — far exceeding PCIe's ~64 GB/s. By contrast, cross-node communication relies on InfiniBand or RoCE (RDMA over Converged Ethernet) networks. Although high-end InfiniBand HDR/NDR can provide link rates of 200–400 Gb/s, actual available bandwidth under multi-hop routing and switch contention is typically one to two orders of magnitude lower than NVLink. This layered bandwidth gap between intra-node and inter-node communication is the fundamental reason topology-aware scheduling delivers value: if two GPUs that need to exchange hundreds of GB of data per second are assigned to different servers, communication overhead multiplies dramatically, creating a clear system bottleneck.

The Core Idea Behind Topology-Aware Scheduling

The essence of topology-aware scheduling is giving the scheduling system "visibility" into the physical layout of the hardware. It needs to understand the connection hierarchy between GPUs: which GPUs reside on the same node, which nodes hang off the same switch, and which span different racks or network domains.

Armed with this topology map, the scheduler can consolidate communication-intensive tasks onto hardware that is physically "closer" together, routing high-frequency communication over high-bandwidth, nearby links and reducing cross-domain traffic. This idea is not new in HPC, but its benefits are significantly amplified in modern AI clusters running thousands of GPUs — the more complex the communication patterns and the larger the scale, the greater the speedup that topology optimization can deliver.

In distributed training, communication load primarily comes from three parallelism strategies: Data Parallelism synchronizes gradients across all nodes, with communication volume proportional to model parameter size; Tensor Parallelism splits individual layer weights across multiple GPUs and requires extremely low-latency all-reduce operations, making it most dependent on NVLink bandwidth; Pipeline Parallelism passes activations between different layers and places relatively lower demands on inter-node links. The practical implication for topology-aware scheduling: GPUs in a tensor parallelism group should be placed on the same node whenever possible (to leverage NVLink), pipeline stages can span nodes but should stay within the same rack, and data parallel replicas can be distributed more flexibly. Different parallelism strategies have different sensitivities to communication topology, and the scheduler needs to understand a workload's communication pattern to make optimal decisions.

Where NVIDIA Topograph Fits In

NVIDIA Topograph is a tool built specifically to address this problem. Its goal is to expose a cluster's network and hardware topology information to the scheduling layer, enabling workloads to be placed intelligently according to topological structure rather than simply assigned to whatever resources happen to be idle.

Combined with inference and scheduling frameworks like NVIDIA Dynamo, Topograph allows operators to automatically factor in the underlying interconnect structure when deploying training or inference jobs. This is especially important for communication-sensitive workloads like large language model inference and distributed training — proper placement can directly translate into lower end-to-end latency and higher cluster throughput.

NVIDIA Dynamo is NVIDIA's distributed inference framework, designed for efficient serving of large language models (LLMs) across multi-GPU, multi-node clusters. Its core capabilities include cross-node KV cache scheduling and reuse, dynamic request routing, and integration with orchestration systems like Kubernetes. Topograph, as the topology information layer, can be thought of as the "map" input for Dynamo's scheduling decisions — Dynamo decides "what to do," while Topograph answers "where it's best done." Together, they enable inference requests to be routed to GPU nodes that are physically closest to already-cached KV data, reducing the extra communication overhead of cache migration. This is particularly impactful for improving throughput in long-context inference scenarios.

What This Means for AI Infrastructure

As model scale continues to grow, the marginal returns from simply piling on more GPUs are diminishing. How to make the most of existing compute has become the new competitive frontier. Topology-aware scheduling represents an evolution in infrastructure optimization — from "resource allocation" to "resource orchestration."

For teams operating AI factories, tools like this mean being able to extract more performance from existing hardware through smarter scheduling, without blindly expanding capacity. In an era of rising power and hardware costs, this kind of software-layer optimization often delivers an exceptionally high return on investment. The emergence of Topograph also reflects NVIDIA's broader strategy of extending its reach upward from chips to build a full-stack software ecosystem spanning scheduling, orchestration, and inference.

Note: This article is based on publicly available information from the NVIDIA Developer Blog. For specific feature details and performance data, please refer to official documentation.