Together GPU Cluster Auto-Repair: From Failure to Healthy Node

In large-scale GPU cluster operations, node failures are the norm rather than the exception. Together GPU Clusters' new auto-repair feature has a clear goal: let failed nodes return to a healthy state automatically whenever possible, while leaving repair decisions in the hands of users.

The Core Logic of Auto-Repair

The system works in three steps: detect, select a plan, execute. It covers three types of failure sources—the GPU itself, the host machine, and the cluster network (fabric). Once a problem is detected, the system does not automatically apply the heaviest repair method, but instead chooses "the lightest fix."

This design choice is worth noting. In traditional operations, a GPU error often directly triggers a node reboot or even a full machine replacement, which is costly and causes long interruptions. Escalating repair measures according to failure severity can reduce disruption to training jobs while still ensuring recovery.

In the context of GPU clusters, fabric refers to the high-speed interconnect network connecting compute nodes, typically including NVIDIA NVLink/NVSwitch (intra-node GPU-to-GPU interconnect) and InfiniBand or RoCE (inter-node communication). Large-scale training relies on collective communication (such as AllReduce) to synchronize gradients across hundreds or thousands of GPUs, so any anomaly on a single link, switch port, or NIC on the fabric may manifest as a hung training job or a sudden drop in throughput rather than a direct error exit. Such failures are harder to locate than single-GPU failures because symptoms often appear at the application layer. Listing fabric as a separate failure source means auto-repair needs to distinguish "GPU compute unit failures" from "interconnect path failures"; the latter may involve resetting links, reinitializing network interfaces, or isolating switch ports—a completely different remediation path from rebooting a GPU.

Fine-Grained Control by Failure Type and Xid Code

The real differentiator of auto-repair is the granularity of control. Users can individually decide whether to enable auto-repair for each type of failure and each Xid error code.

Xid is NVIDIA's standard code system for GPU driver errors, with different codes corresponding to different kinds of hardware or driver issues. Pushing repair policies down to the Xid level means operations teams can allow only error types known to be safely handled automatically, while keeping manual intervention for codes that may indicate hardware degradation and require human confirmation. This "cautious by default, authorize item by item" approach is closer to real production needs than one-size-fits-all automation.

Xid messages are written to the system log by the NVIDIA driver when it detects a GPU anomaly (typically appearing in dmesg or /var/log/syslog on Linux), in a format like NVRM: Xid (PCI:0000:3b:00): 79, pid=12345. The number itself points to the problem category: for example, Xid 13/31 are mostly related to illegal memory access or user application out-of-bounds access, Xid 48 usually indicates a double-bit ECC error, Xid 79 means the GPU has fallen off the bus, and Xid 62/64 are often related to page table or internal microcontroller anomalies. Recoverability varies greatly across codes—some are triggered by application bugs and can be recovered by restarting the process or node; others foretell hardware degradation in memory or power delivery, and repeated auto-repair would instead mask the problem. This is exactly the significance of per-Xid configuration switches: operations teams can define which codes allow automatic handling based on their own hardware batches and historical failure data.

What It Means for Cluster Operations

The availability bottleneck of GPU clusters often lies not in single-GPU performance but in the response speed of failure recovery. Auto-repair shortens the path from failure occurrence to a node becoming available again, reducing back-and-forth manual troubleshooting. At the same time, policies configured by failure type keep the automation boundary clear and controllable, preventing auto-repair from masking underlying systemic problems.

For teams running large-scale training jobs, the value of this capability lies in freeing operations staff from repetitive failure handling and focusing them on anomalies that truly require judgment. The information Together has released this time focuses on the mechanism itself; the specific list of repair actions and the scope of Xid coverage still await more detailed documentation.