Running a 141B LLM on a Home PC Cluster: Glacier Distributed Inference Hands-On

Title

Running a 141B LLM on a Home PC Cluster: Glacier Distributed Inference Hands-On

Meta Description

The Glacier.Inference.Distributed pipeline parallelism uses pure C# to split large models like Qwen2-57B and Mixtral-8x22B across multiple home PCs, with gigabit network latency accounting for only 0.2%. Includes a 4-node setup tutorial and benchmark data.

Article Content

The most frustrating barrier to running large models locally isn't the algorithm—it's VRAM. A single consumer-grade GPU typically has between 8GB and 16GB of VRAM, and squeezing models like Qwen2-57B or Mixtral-8x22B (141B parameters) into one machine often requires a data center accelerator with 80GB of VRAM. But if you happen to have a few machines of varying specs scattered around your desk—a desktop with an RTX 3060, a laptop with an RTX 4060, a mini PC with an AMD Radeon 680M integrated GPU, and a Linux machine that supports AVX-512—can they be stitched together into an inference cluster? The Glacier.Inference.Distributed project on GitHub is aimed squarely at this scenario.

Core Idea: Heterogeneous Pipeline Parallelism

Glacier.Inference.Distributed doesn't require uniform hardware specs across the cluster, nor does it need a complex distributed training framework. It uses pipeline parallelism to split a GGUF-format Transformer model by layers across N machines.

The hidden state activations for a single token flow through the entire cluster in sequence: Node 0 handles the embedding layer and initial attention blocks, Node 1 receives and computes the middle layers, Node 2 continues further, and Node 3 completes the final layers and outputs the LM Head and logits, ultimately producing the next token. Each machine only retains the layers it's responsible for, and weights are never transmitted over the network.

The key to this design is that it tolerates hardware heterogeneity. An RTX 3060 node can carry more layers, an integrated GPU node takes fewer, and a CPU node uses the AVX-512 SIMD path. The coordinator automatically inspects the GGUF metadata and computes layer boundaries and hidden dimensions based on the nodes you provide.

Is Gigabit LAN Really the Bottleneck?

A common intuition is that using ordinary gigabit Ethernet for distributed inference would drag down token generation speed due to network latency. But actual measurements give the opposite conclusion.

During the sequential decoding phase of an LLM, what the cluster exchanges is not model weights but activation tensors (hidden state vectors). Taking hidden dimension d=3584 as an example: at FP32 precision that's about 14.3KB per token, and after dynamic quantization to symmetric INT8 it's only 5.1KB per hop on the network. Under standard gigabit, one hop is about 0.45ms, and a full 3-hop trip is about 7.88ms.

By comparison, multi-head attention and MoE feed-forward network computation for a heavy model requires hundreds of milliseconds to seconds. That ~8ms of network latency accounts for less than 0.2% of total token execution time, with the remaining 99.8% spent on raw computation. This means gigabit is more than sufficient—the bottleneck isn't the network.

The difference between pipeline parallelism and tensor parallelism: Pipeline parallelism slices the model along its "depth" direction—each node handles a consecutive set of layers, and a token's activations must pass through all nodes in sequence, so only one node is computing the current token at any given moment (unless micro-batch pipeline overlap is used). Tensor parallelism, by contrast, splits the weight matrices of the same layer across multiple machines; each layer computation requires an All-Reduce communication, making it extremely sensitive to bandwidth and latency, and it's typically only practical under high-speed interconnects like NVLink. There's also data parallelism, which replicates the entire model and splits the input data—suitable for training but unable to solve the problem of a model not fitting on a single machine. Glacier chooses pipeline parallelism precisely because it has the lowest network bandwidth requirements and naturally tolerates differences in node compute power—slow nodes can take fewer layers, which is exactly what heterogeneous home clusters need most.

Sequential decoding: LLM inference has two phases. The prefill phase forwards the entire prompt in parallel at once, is matrix-multiplication intensive, saturates the GPU, and determines first-token latency; the decode phase generates autoregressively token by token, processing only one token per step, so the computation shrinks dramatically and the bottleneck shifts from compute to memory access and communication. Sequential decoding refers to the latter. Under pipeline parallelism, each token in the decode phase must pass an activation along the node chain and wait for the return, so the per-token communication latency gets added to the total time. Glacier uses INT8 dynamic quantization to compress each transmission to the ~5KB level and uses 64-byte-aligned fused frames to reduce system calls—precisely because the constant overhead of this step gets multiplied directly by the number of generated tokens, making it the key to whether distributed decode is usable.

Zero Dependencies: Pure C# .NET Implementation

Most distributed AI stacks drag along a pile of heavy dependencies: CUDA toolchains, Python runtimes, Torch binaries, native C++ orchestration layers. Glacier.Inference.Distributed, in contrast, is built entirely from scratch in pure C# targeting .NET 10, with Native AOT support.

A few design points worth noting:

• Zero Python, zero C++ dependencies: No need to configure a Python environment on each machine during deployment
• Network pipeline: Built on System.IO.Pipelines, using 64-byte-aligned socket frames (ClusterFrameHeader + FusedActivationHeader)
• Memory invariants: 0 Gen0/Gen1/Gen2 garbage collections in the hot decode loop, with 0 bytes of managed heap allocation
• Hardware acceleration: Automatically routes computation to discrete GPUs (Direct3D 12), integrated APUs, or multi-threaded SIMD/AVX-512 vector paths on CPU

This design is very friendly to home clusters—you don't need to unify the software environment across all machines just to run inference.

Native AOT and zero allocation on the hot path: Native AOT (Ahead-Of-Time) means compiling IL directly into native machine code at build time, producing a single-file executable that needs no runtime JIT—fast startup, low memory footprint, and no need to install the .NET runtime on each machine, which matters a lot for the "copy the program onto an old laptop and run it" scenario. Meanwhile, "0 GC in the hot decode loop" depends on two things: first, all per-token temporary buffers are reused on the stack or in pre-allocated object pools, generating no managed heap allocations; second, it uses ref struct, Span<byte>, and ArrayPool to manage tensor slices, avoiding triggering Gen0 collections. Once a GC triggers mid-token-generation, it causes millisecond-level pauses that show up directly as user-visible stutter.

Which Models Can Be Split: Size Reference

After automatically inspecting standard GGUF metadata, the coordinator automatically computes layer boundaries and hidden dimensions based on the nodes. Here is the official reference data:

Model Architecture	Parameters	Per-token Activation	4-node Split Example	Disk Usage
Qwen2-57B-A14B	57B	14B (8/64 MoE)	7 layers per node (28 total)	~25.6 GB (Q3_K_M)
Mixtral-8x22B	141B	39B (2/8 MoE)	14 layers per node (56 total)	~35.3 GB (IQ2_XXS)
Mixtral-8x7B	47B	13B (2/8 MoE)	8 layers per node (32 total)	~26.4 GB (Q4_K_M)
DeepSeekAd-R1-Distill-14B	14.7B	14.7B (Dense)	12 layers per node (48 total)	~9.0 GB (Q4_K_M)

As you can see, the quantized disk usage is not demanding for home storage—Mixtral-8x22B needs only about 35GB even with IQ2_XXS.

Hands-On: Building a 4-Node Home Cluster

The deployment process has four steps.

1. Clone the Repository on All Nodes

First install the .NET 10 SDK, then clone the repository:

git clone https://github.com/ian-cowley/Glacier.Inference.Distributed.git
cd Glacier.Inference.Distributed

2. Download the Model

Glacier has a built-in pure-C# multi-stream HTTP Range downloader that opens concurrent sockets to saturate bandwidth, with no reliance on huggingface-cli or external Python tools:

# Download Qwen2-57B
dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- download qwen2-57b

# Or download Mixtral-8x22B (141B)
dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- download mixtral-8x22b

3. Start the Worker Daemons

Start workers on the secondary machines. Each worker automatically detects whether to initialize D3D12 GPU shaders or CPU SIMD:

# Machine 1 (192.168.1.108, RTX 3060)
dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- --role worker --port 5055

# Machine 2 (192.168.1.101, Radeon 680M)
dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- --role worker --port 5056

# Machine 3 (192.168.1.200, Linux CPU)
dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- --role worker --port 5057

4. Start the Coordinator

On the primary machine, point to all worker endpoints:

dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- \
  --role coordinator \
  --stages 192.168.1.108:5055,192.168.1.101:5056,192.168.1.200:5057 \
  --model ./models/qwen2-57b-a14b-instruct-q3_k_m.gguf \
  --tokens 50 \
  --prompt "Mixture-of-Experts enables scaling parameter capacity across distributed home clusters by"

The coordinator connects over TCP, verifies hardware status, assigns layer boundaries, and streams tokens directly to the console.

Embedding Distributed Inference in C# Code

If the CLI tool isn't enough, the coordinator can also be embedded into .NET applications via a simple async streaming API:

using Glacier.Inference.Distributed;

var endpoints = new[]
{
    new StageEndpoint("192.168.1.108", 5055, "RTX 3060 Node"),
    new StageEndpoint("192.168.1.101", 5056, "APU Node"),
    new StageEndpoint("192.168.1.200", 5057, "AVX-512 Server")
};

using var coordinator = new DistributedPipelineCoordinator(
    stages: endpoints,
    modelPath: "models/Mixtral-8x22B-Instruct-v0.1.i1-IQ2_XXS.gguf");

await foreach (var token in coordinator.GenerateStreamAsync("Explain distributed tensor pipelines in simple terms."))
{
    Console.Write(token);
}

This embedding approach is well suited to integrating distributed inference into existing .NET services.

Real Physical Cluster Benchmarks

The project tested DeepSeek-R1-Distill-Qwen-14B on 4 physical machines connected via a basic 1Gbps switch, yielding the following measured metrics:

• Prompt prefill speed: 52.46 tok/s (6.19x faster than baseline via batched SIMD FMA)
• Speculative decoding speed: 23.24 tok/s, with 2-stage async pipeline overlap
• Network latency: 0.45ms per hop, using fused 64-byte binary frames
• Network precision: Compressed from 20.4KB (FP32) to 5,124 bytes (INT8) per token, with no activation drift
• Memory pressure: 0 Gen0/Gen1/Gen2 collections, guaranteeing no GC pauses on the hot path

These figures come from a single project's self-testing and don't mean every hardware combination can reproduce them, but they at least show that running a 14B-class model over ordinary gigabit and heterogeneous hardware is feasible.

Prefill throughput and speculative decoding: The 52.46 tok/s in the table refers to the equivalent token throughput of the prefill phase (processing the entire prompt at once); since all tokens within the prompt can be computed in parallel, this figure is far higher than the token-by-token decode speed. The project notes its batched SIMD FMA is 6.19x faster than baseline, indicating this value mainly reflects compute rather than network. The 23.24 tok/s for speculative decoding, on the other hand, is the decode phase speed: it first uses a small "draft model" to guess several candidate tokens at once, then the main model verifies them in a single parallel batch, directly accepting those it approves—thereby merging multiple originally serial forward passes into one. This neatly sidesteps the "one round trip per token" latency bottleneck of pipeline parallelism, and combined with 2-stage async pipeline overlap, it lets communication and computation proceed as much in parallel as possible.

Putting Idle Hardware Back to Work

Large models shouldn't belong only to cloud APIs or high-end enterprise workstations. Glacier.Inference.Distributed's approach is to combine lean, zero-allocation network frames with modern SIMD and graphics APIs to aggregate idle hardware on the local network into a cluster capable of running large models.

This project is best suited for developers or enthusiasts who have 2 to 4 machines of varying configurations at home. If all your devices are the same model and spec, it might be simpler to just buy a card with more VRAM; but if you happen to have a pile of scattered old machines, laptops, and mini PCs, stringing them together to run models that would never fit on a single machine is indeed an interesting direction for reuse.

Full source code, benchmarking tools, and documentation are on GitHub: github.com/ian-cowley/Glacier.Inference.Distributed.

One-line Summary

A pure C# project uses pipeline parallelism to split GGUF large models across heterogeneous home PC clusters—gigabit network is enough.

Paragraph Summary

Glacier.Inference.Distributed is an open-source project written in pure C# .NET 10, aiming to stitch together a few idle home computers of varying configurations (such as an RTX 3060 desktop, a Radeon 680M mini PC, and an AVX-512 Linux machine) into an inference cluster capable of running large models. It uses pipeline parallelism to split a GGUF model by layers, with each machine retaining only its own layers, and activation tensors passing through the nodes in sequence. The project demonstrates that gigabit is not the bottleneck: after INT8 quantization each token is only about 5KB per hop, and a 3-hop total is about 7.88ms, less than 0.2% of total execution time. The implementation has zero Python and zero C++ dependencies, 0 GC and 0 bytes of managed heap allocation in the hot decode loop, and automatically routes to D3D12 GPU or CPU SIMD. Official 4-node tests of DeepSeek-R1-Distill-14B reached 52.46 tok/s prefill and 23.24 tok/s speculative decoding.