Betelgeuse
An inference engine for large MoE models based on a fork of llama.cpp. Through zero-gating pruning and memory optimization techniques, it enables efficient inference of large models on consumer-grade hardware with limited VRAM and memory while guaranteeing bit-identical output.
Timeline (last 90 days)
The author is porting Betelgeuse's streaming + victim cache architecture to DeepSeek V4.1 Flash (1017), with early tests already reaching about 25 tok/s on comparable hardware; the project is open-sourced on GitHub (Betelgeuse-AI/Betelgeuse) and offers a pre-packaged Alpha-Orionis-XS bundle on HuggingFace.
Betelgeuse strips the expert payload from Shard 1, making the main files only 3.2GB (Dense.gguf) + 35.3GB (experts.bin, 4KB-aligned single read blocks) + 26.8GB (PLE.gguf n-gram shards), avoiding keeping 35GB of dead expert tensors in each of the Shard 1 .gguf and experts.bin.
On the Q8_K_XL quantization at about 190GB total size, bel-server achieves 8 tok/s cold and 12 tok/s warm (peaking at 21 tok/s) on an RTX 3060 12GB + 16GB DDR4 machine, whereas stock llama.cpp immediately OOMs or crawls at <0.1 tok/s.
The Betelgeuse inference engine can continuously run the 68GB Qwen 3.8 Flash Next MoE model at 22 tok/s on a PC with an RTX 3060 12GB graphics card and 16GB DDR4 memory, and the output is 100% bit-identical to the original model.
Betelgeuse is a fork of llama.cpp (bel-cli / bel-server), and the author is also submitting PRs upstream.
In Betelgeuse, all 10 routed experts of every layer are computed at full precision, and the author verified byte-for-byte against the original stock output using 28,080 routing records and 576 generated tokens, with a difference of 0.000e+00.
Betelgeuse keeps the approximately 3.2GB dense layers + GDN + QSA + LM head in VRAM, and the remaining VRAM holds about 5,000 expert slots, distributed as: 240 slots per layer for early layers 0–4, 92 slots for middle layers, and 60 slots for tail layers.
Betelgeuse uses an in-stream CUDA kernel to resolve expert hits on-device in about 2µs, notifying the host via a doorbell only when an expert is missing, so that the entire decoding process runs as a single CUDA Graph replay, avoiding the roughly 360ms cudaStreamSynchronize overhead per token.
Betelgeuse uses a RAM victim cache plus an io_uring pager to reduce physical NVMe reads from about 71 MB/token to 12.43 MB/token.
Betelgeuse is a red supergiant.
1 more timeline events
All Facts (11)
Betelgeuse strips the expert payload from Shard 1, making the main files only 3.2GB (Dense.gguf) + 35.3GB (experts.bin, 4KB-aligned single read blocks) + 26.8GB (PLE.gguf n-gram shards), avoiding keeping 35GB of dead expert tensors in each of the Shard 1 .gguf and experts.bin.
50%UnverifiedOn the Q8_K_XL quantization at about 190GB total size, bel-server achieves 8 tok/s cold and 12 tok/s warm (peaking at 21 tok/s) on an RTX 3060 12GB + 16GB DDR4 machine, whereas stock llama.cpp immediately OOMs or crawls at <0.1 tok/s.
50%UnverifiedThe Betelgeuse inference engine can continuously run the 68GB Qwen 3.8 Flash Next MoE model at 22 tok/s on a PC with an RTX 3060 12GB graphics card and 16GB DDR4 memory, and the output is 100% bit-identical to the original model.
50%UnverifiedBetelgeuse is a fork of llama.cpp (bel-cli / bel-server), and the author is also submitting PRs upstream.
50%UnverifiedIn Betelgeuse, all 10 routed experts of every layer are computed at full precision, and the author verified byte-for-byte against the original stock output using 28,080 routing records and 576 generated tokens, with a difference of 0.000e+00.
50%UnverifiedBetelgeuse keeps the approximately 3.2GB dense layers + GDN + QSA + LM head in VRAM, and the remaining VRAM holds about 5,000 expert slots, distributed as: 240 slots per layer for early layers 0–4, 92 slots for middle layers, and 60 slots for tail layers.
50%UnverifiedBetelgeuse uses an in-stream CUDA kernel to resolve expert hits on-device in about 2µs, notifying the host via a doorbell only when an expert is missing, so that the entire decoding process runs as a single CUDA Graph replay, avoiding the roughly 360ms cudaStreamSynchronize overhead per token.
50%UnverifiedBetelgeuse uses a RAM victim cache plus an io_uring pager to reduce physical NVMe reads from about 71 MB/token to 12.43 MB/token.
50%UnverifiedBetelgeuse is a red supergiant.
50%UnverifiedThe author is porting Betelgeuse's streaming + victim cache architecture to DeepSeek V4.1 Flash (1017), with early tests already reaching about 25 tok/s on comparable hardware; the project is open-sourced on GitHub (Betelgeuse-AI/Betelgeuse) and offers a pre-packaged Alpha-Orionis-XS bundle on HuggingFace.
50%UnverifiedA post appeared on Reddit claiming that OpenAI is testing a model named GPT-6.2 Betelgeuse with the internal codename Bel, and the information originated from a leak by a user on platform X.
50%