One GPU Isn't Enough: Deploying GLM-5.3 Across Two DGX Nodes with vLLM

Large Models That Don't Fit on One Machine Are Forcing a Change in Deployment

Running a frontier-scale model locally requires that all model weights stay resident in GPU memory. Once the parameter count exceeds what a single machine can hold, the traditional "one machine, one model" approach breaks down. NVIDIA's developer blog offers a concrete example with GLM-5.3: it can no longer fit on a single device and must span two DGX Stations, with vLLM as the core component enabling this setup.

The significance here isn't any particular deployment demo — it's the trend it reveals: model scale is growing faster than single-machine GPU memory capacity. Multi-node distributed inference has shifted from "optional" to "mandatory," and engineering teams need to build cross-node deployment capabilities ahead of time.

Weight Residency Is a Hard Constraint — Sharding Is the Only Way Out

The first physical barrier in LLM inference is fitting all post-training weights into GPU memory. GLM-5.3 falls short on this line.

The only option is sharding. In vLLM's approach, weights can be split across the memory of different devices, and tensor parallelism lets multiple GPUs collaboratively complete a single forward pass. When scaling from one machine to two, the sharding dimension extends from GPU-to-GPU within a node to GPU-to-GPU across nodes, and communication overhead and synchronization complexity rise accordingly.

It's worth noting this isn't simply "splitting the model in half." Cross-node tensor parallelism requires high-bandwidth, low-latency interconnect between the two nodes; otherwise, the communication bottleneck will swallow the gains from parallelism. The network configuration between DGX Stations thus becomes a critical variable in the whole scheme, not a negligible detail.

Tensor Parallelism is the most common sharding strategy in multi-node inference. It works by splitting large weight matrices in the model into blocks along specific dimensions, with each GPU holding one block. During forward computation, devices execute local matrix multiplications in parallel, then aggregate results through collective communication operations like All-Reduce. Unlike pipeline parallelism, which splits by layer, tensor parallelism unfolds within each layer, placing higher demands on inter-device bandwidth. Within a single machine, NVLink's high bandwidth supports this communication pattern well; once across machines, communication must traverse network switches, significantly increasing latency — this is the core bottleneck facing cross-node tensor parallelism.

Weight residency means model weights must be fully stored in GPU memory during inference, rather than loaded on demand from host memory or disk. This is because a single forward pass through a large model requires traversing all weights; if weights aren't in GPU memory, frequent data transfers introduce unacceptable latency. Therefore, the comparison between total model weight size and available single-machine GPU memory directly determines whether single-machine deployment is possible — which is the fundamental reason GLM-5.3 must be deployed across machines.

What Role Does vLLM Play in Multi-Node Deployment?

vLLM has become the default choice for such deployments because it encapsulates the most repetitive engineering work in distributed inference. In multi-node scenarios, the problems to solve include: how to distribute tensor parallelism across two nodes, how to coordinate node startup order, and how to correctly route requests across GPUs on both machines. All these considerations — unnecessary in single-machine deployment — surface in dual-machine setups.

vLLM consolidates these processes into a relatively unified configuration interface, so developers don't have to write cross-node communication logic from scratch. For teams wanting to quickly validate multi-node deployment feasibility, this encapsulation significantly lowers the startup cost. The trade-off: once the topology or network conditions deviate from defaults, tuning still requires detailed investigation at the level of vLLM's parallelism configuration parameters.

vLLM is a high-throughput inference engine for large language models. Its core innovation is PagedAttention — borrowing from OS virtual memory paging to manage the KV Cache, storing attention key-value cache in blocks, significantly reducing memory fragmentation and improving concurrent request handling efficiency. In multi-node scenarios, vLLM coordinates worker processes across nodes through distributed runtimes like Ray, establishes tensor parallel groups across machines, and exposes parameters such as parallelism degree and node roles through a unified configuration interface. This design — encapsulating distributed communication and scheduling logic into the framework — makes vLLM the de facto starting point for cross-machine deployment; but encapsulation also means that when network topology or hardware conditions exceed defaults, tuning requires digging into its parallelism configuration and runtime parameters.

The Real Trade-offs of Dual-Machine Deployment

Spreading the model across two machines solves the "doesn't fit" problem, but it isn't free.

• Communication overhead becomes a new cost. Every cross-node forward pass involves inter-node data exchange. NVLink-level bandwidth within a machine can't be replicated across machines, and increased latency is unavoidable.
• Resource utilization is no longer a single dimension. GPU memory on both nodes is occupied simultaneously, compressing the room for elastic scheduling; deployment planning must consider both machines' resources as a whole.
• Operational complexity increases. The failure domain expands from one node to two; any anomaly on either side affects the availability of the entire inference service.
• The impact of misconfiguration is amplified. Parameters like parallelism degree, node roles, and network addresses are coupled in dual-machine scenarios, making debugging paths for misconfigurations far longer than on a single machine.

These trade-offs don't justify rejecting cross-machine deployment — model scale dictates that there's no simpler alternative. What they really signal is that multi-node inference requires stricter pre-deployment evaluation, not simply copying a single-machine script to a second machine.

What This Means for Local Deployers

The signal from NVIDIA's demo is clear: local deployment of frontier models is moving from the single-machine era into the multi-machine era.

For teams already equipped with DGX Stations or similar hardware, building cross-node inference capabilities early is the pragmatic choice. Weight sharding strategy, parallelism configuration, and inter-node network quality directly determine whether multi-node deployment can achieve acceptable performance. For those still planning hardware, it's worth revisiting whether the assumption "one machine is big enough" still holds — model scale growth hasn't stopped, and the gap between GPU memory capacity and model size will only continue to widen.

GLM-5.3 is just one concrete case, but the problem it points to is universal: when a model doesn't fit on one machine, the deployment architecture itself must become a design object.