Glacier.Inference.Distributed
An open-source distributed LLM inference project implemented purely in C# .NET. It uses pipeline parallelism to split GGUF models layer by layer across multiple heterogeneous devices, supports Native AOT, and has zero Python or C++ dependencies.
Timeline (last 90 days)
Glacier.Inference.Distributed adopts pipeline parallelism, splitting a GGUF-format Transformer model by layer across N machines, and does not require uniform cluster hardware specifications.
During the LLM sequential decoding stage, a pipeline-parallel cluster exchanges activation tensors rather than model weights; taking hidden dimension d=3584 as an example, at FP32 precision this is about 14.3KB per token, and after symmetric INT8 dynamic quantization it is only 5.1KB per hop on the network.
On standard gigabit Ethernet, one pipeline-parallel hop is about 0.45ms, and a full 3 hops is about 7.88ms, accounting for less than 0.2% of total token execution time, so gigabit Ethernet is completely sufficient for distributed inference, and the bottleneck is not the network.
Glacier.Inference.Distributed is built entirely from scratch in pure C# targeting .NET 10, supports Native AOT, has zero Python and zero C++ dependencies, and does not require configuring a Python environment on each machine during deployment.
Glacier.Inference.Distributed's network pipeline is based on System.IO.Pipelines, using 64-byte-aligned socket frames (ClusterFrameHeader+FusedActivationHeader).
In Glacier.Inference.Distributed's hot decoding loop, 0 Gen0/Gen1/Gen2 garbage collections are achieved, with 0 bytes of managed heap allocation.
Glacier.Inference.Distributed can automatically route computation to a discrete GPU (Direct3D 12), an integrated APU, or a multithreaded SIMD/AVX-512 vector path on the CPU.
The Glacier coordinator splits Qwen2-57B-A14B (57B parameters, 14B activated per token, 8/64 MoE) across 4 nodes at 7 layers per node for a total of 28 layers, with a disk footprint of about 25.6GB (Q3_K_M quantization).
Glacier has a built-in pure C# multi-stream HTTP Range downloader that opens concurrent sockets to saturate bandwidth and does not depend on huggingface-cli or external Python tools.
The Glacier.Inference.Distributed project is located at the GitHub repository https://github.com/ian-cowley/Glacier.Inference.Distributed.git, and deployment requires first installing the .NET 10 SDK.
All Facts (10)
Glacier.Inference.Distributed adopts pipeline parallelism, splitting a GGUF-format Transformer model by layer across N machines, and does not require uniform cluster hardware specifications.
50%UnverifiedDuring the LLM sequential decoding stage, a pipeline-parallel cluster exchanges activation tensors rather than model weights; taking hidden dimension d=3584 as an example, at FP32 precision this is about 14.3KB per token, and after symmetric INT8 dynamic quantization it is only 5.1KB per hop on the network.
50%UnverifiedOn standard gigabit Ethernet, one pipeline-parallel hop is about 0.45ms, and a full 3 hops is about 7.88ms, accounting for less than 0.2% of total token execution time, so gigabit Ethernet is completely sufficient for distributed inference, and the bottleneck is not the network.
50%UnverifiedGlacier.Inference.Distributed is built entirely from scratch in pure C# targeting .NET 10, supports Native AOT, has zero Python and zero C++ dependencies, and does not require configuring a Python environment on each machine during deployment.
50%UnverifiedGlacier.Inference.Distributed's network pipeline is based on System.IO.Pipelines, using 64-byte-aligned socket frames (ClusterFrameHeader+FusedActivationHeader).
50%UnverifiedIn Glacier.Inference.Distributed's hot decoding loop, 0 Gen0/Gen1/Gen2 garbage collections are achieved, with 0 bytes of managed heap allocation.
50%UnverifiedGlacier.Inference.Distributed can automatically route computation to a discrete GPU (Direct3D 12), an integrated APU, or a multithreaded SIMD/AVX-512 vector path on the CPU.
50%UnverifiedThe Glacier coordinator splits Qwen2-57B-A14B (57B parameters, 14B activated per token, 8/64 MoE) across 4 nodes at 7 layers per node for a total of 28 layers, with a disk footprint of about 25.6GB (Q3_K_M quantization).
50%UnverifiedGlacier has a built-in pure C# multi-stream HTTP Range downloader that opens concurrent sockets to saturate bandwidth and does not depend on huggingface-cli or external Python tools.
50%UnverifiedThe Glacier.Inference.Distributed project is located at the GitHub repository https://github.com/ian-cowley/Glacier.Inference.Distributed.git, and deployment requires first installing the .NET 10 SDK.
50%