Running Large Models on Consumer Hardware: The Tug-of-War Between Distributed Deployment and Memory Budget

When you run large language models on consumer hardware, the real bottleneck is often not the model itself, but memory. This is the fifth article in the "LLMs on Consumer Hardware" series, telling a realistic story about memory contention: when the coordinator model and the worker model are crammed onto the same machine, the system nearly collapses; but through "distributed deployment" and "sensible downsizing," the problem is solved cleanly.

From Memory Contention to a Distributed Breakthrough

The fourth article in the series ended by hitting a wall: the 77GB coordinator model gpt-oss:120b, which was worth running, could not share a single machine with the worker models it invoked. Every task delegation triggered roughly 100GB of model swapping, and even a simple greeting took nearly five minutes. The conclusion was clear: the coordinator and worker models must reside in memory simultaneously.

The solution in Part 5 is straightforward: stop making them share. The author migrated the worker models to a second machine—a 5600G host equipped with a GTX 1060. This way, the main machine's memory was no longer contested, the coordinator had all memory to itself, and it would not be evicted by local processes during handoffs.

The effect was immediate. The same 120B model that had been unusable due to frequent swapping now loaded once and stayed resident throughout a long run of 1 hour and 46 minutes, with no swapping at all. Once freed from the swapping penalty, its orchestration reliability even surpassed Gemma. The lesson was almost frustratingly simple: the coordinator was never the problem—its location was.

This second host delivered dual value. It was introduced in Part 2 because its dedicated GPU could solve the prefill problem—machines without a GPU cannot ingest large prompts in a reasonable amount of time. Moving the worker models there solved another, unrelated problem: it freed up the memory the coordinator needed on the main machine. One hardware decision solved two constraints.

A New Tenant Brought by an Idle GPU

This arrangement had one key side effect: the 120B model far exceeded the 16GB of VRAM, so it ran mainly on the CPU and system memory—slow, at about 9 tokens per second, but the GPU was almost entirely idle during inference. The high-performance graphics card on the main machine had nothing to do.

That idle GPU was an invitation. The author added local image generation on the main machine's 6900XT (ComfyUI with Stable Diffusion), enabling the agent to generate images, not just text and code. Running on its own, it was fast—roughly 15 to 30 seconds for a single SDXL image.

The problem was that the main machine now had two heavyweight tenants. The coordinator occupied about 77GB of the ~80GB of memory, and image generation needed the GPU plus a few GB of memory. The GPU was idle—the coordinator wasn't using it—but memory once again became the bottleneck.

ComfyUI's /free Endpoint and the VRAM Baseline

ComfyUI is a node-graph-based Stable Diffusion frontend that allows text encoding, sampling, and decoding steps to be composed through API workflows. It provides a /free endpoint that can actively unload loaded model weights, clear cached tensors in VRAM, and trigger Python garbage collection.

By default, after generating an image, the SDXL checkpoint, CLIP text encoder, and VAE remain resident in VRAM, occupying about 8GB. After calling /free, the model weights are released, and VRAM drops to a baseline of only about 2.2GB, consisting of the CUDA runtime and a small amount of working buffers. This mechanism allows the image generator to use almost zero VRAM when not in use, enabling it to coexist with a CPU-inference coordinator that also relies on system memory—provided that system memory itself has headroom. This strategy reduces the VRAM peak, but it cannot reduce system memory demand: loading an image model still requires allocating several GB of RAM instantaneously.

Coexistence Experiment: A Seemingly Plausible Hypothesis

It was worth testing rather than guessing, because there was a seemingly plausible argument supporting coexistence. During task delegation, the coordinator is idle: it has handed the task to a remote worker model and is waiting. Idle memory pages can be swapped out cheaply, and only actively used memory causes thrashing. In principle, Linux should be able to swap out the waiting coordinator, let image generation use the freed memory, and then swap it back in one go when the coordinator resumes. An idle tenant should be a good neighbor.

Two safeguards were set up before the test: a memory guard for the image generator that refuses to start if free memory falls below 8GB—failing cleanly rather than driving the machine into a swap-induced freeze; and automatic unloading after each image via ComfyUI's /free, which truly releases the model and reduces VRAM from about 8GB to a 2.2GB baseline. Then the guard was disabled, the 120B model was loaded, a multi-image run was started, and free -h and vmstat were monitored simultaneously.

The hypothesis failed. With the 120B resident, only about 3GB of memory was free, which was far from enough.

**Memory Thrashing and Swapping Out Idle Pages**

"Thrashing" refers to the situation where the system's physical memory is insufficient to hold all active working sets simultaneously, forcing the kernel to frequently swap pages between memory and swap space. At this point, the CPU and GPU spend a great deal of time waiting on I/O rather than performing inference, and actual throughput drops sharply, manifesting as a response that is "graceful but extremely slow."

In theory, Linux can cheaply swap out memory pages of idle processes: the coordinator is blocked while waiting for the remote worker model to return, so most of its resident pages can be marked reclaimable, yielding memory to image generation. The problem is that if the total amount of simultaneously resident models is already approaching the physical memory limit, the kernel has very little room to reclaim—each GPU generation needs several GB, and freeing that memory means some pages of the coordinator must be swapped out; after generation finishes, the coordinator must be swapped back in to resume. The volume of swap-in and swap-out far exceeds the actual computation, so "cheap" becomes "continuous," which is precisely why in the experiment the first image completed and all the rest timed out.
# Main machine, 120B coordinator resident (~77GB), multi-image run:
free -h:   ~3 GB free
image 1:   completed
image 2:   timed out at 300s  ->  placeholder
image 3:   timed out at 300s  ->  placeholder
   ...

The first image completed, and the rest timed out at the 300-second limit, falling back to placeholders. The elegant argument about idle pages was not wrong in principle—it was simply overwhelmed in practice. Under such heavy memory pressure, every GPU generation took more than five minutes, and "cheap swapping" became "continuous swapping." The machine degraded gracefully rather than crashing, but its capability was effectively blocked.

Sensible Downsizing: Less Is More

The solution was not more hardware, but fewer models. The 120B had always been on the edge: it consumed almost the entire memory budget by itself, which made both the original worker-model swapping and the current image-generation conflict unavoidable. Downgrading the coordinator to gpt-oss:20b, about 14GB, left about 60GB free—enough to coordinate and generate images at the same time, with no swapping and no guard refusals.

That trade looked worse than it actually was. The 120B's only real advantage was more reliable orchestration, and—as later articles show—even that did not solve the deeper reliability problem it was supposed to solve. A coordinator that consumed an entire machine to run and still failed to solve the hardest problem was hard to justify over a model one-fifth its size.

The Memory Budget: The Core Law of Local Multi-Model Systems

The throughline of this article is a single idea: local multi-model systems are governed by the memory budget. The question is never whether a single model fits, but whether everything that must be resident at the same time can fit together—the coordinator, the worker models, and now the image generator. There are two levers for controlling this budget, and both beat buying a larger model: placement (which machine things run on) and sensible downsizing (how large each model needs to be to do its job). Distributing the worker models is placement; shrinking the coordinator is sensible downsizing. Together, they made the memory contention from the previous two articles disappear.

Next Step

With the memory problem solved, a reasonable hypothesis still held: a larger coordinator, now that it could finally run, should be better. The next article directly tests that hypothesis—and finds that the larger model buys more reliable ordering, but not a reliable loop, and that loops need something other than a model at all.

For hobbyists tinkering with local AI on consumer hardware, the lesson from this series is direct: before chasing larger parameter counts, examine your memory budget and task placement. Sometimes, changing machines or downgrading a model solves the problem better than upgrading hardware.

One-line Summary

At the core of local multi-model systems is the memory budget; distributed placement and sensible downsizing beat hardware upgrades.

Paragraph Summary

This is the fifth article in the "LLMs on Consumer Hardware" series, telling a story of memory contention. The 77GB gpt-oss:120b coordinator could not coexist on one machine with its worker models, with each delegation triggering about 100GB of swapping and a simple greeting taking nearly five minutes. The solution was to move the worker models to a second host, giving the coordinator exclusive memory. The effect was immediate: the model stayed resident for nearly two hours with no swapping. The second host's idle GPU was then used for image generation, but memory again became the bottleneck—the coexistence experiment failed, with the first image succeeding and all subsequent ones timing out. Ultimately, downgrading the coordinator from 120B to 20B freed about 60GB, enabling coordination and image generation to coexist. From this, the article distills a core law: local multi-model systems are governed by the memory budget, and the two levers of placement and downsizing are better than buying a larger model.