Colibri Pure-C Engine Goes Viral: Running a 744B Model on 25GB of RAM

Why Frontier Models Are Stuck in Data Centers

For a long time, running a 744B-parameter MoE model meant only one thing: rack servers, stacked GPUs, and cloud API calls. The reason is straightforward — a 744B-scale MoE model's full weights run into hundreds of gigabytes, while consumer GPUs top out at 16–24GB of VRAM. Fitting the entire model into VRAM is simply out of the question.

Three pain points block ordinary users. First, the model won't fit — there's never enough VRAM. Second, the weights aren't in your hands — using a cloud API means you can't modify or revoke access, and when the connection drops, the service stops. Third, inference frameworks are too heavy — common runtimes require a full Python and CUDA stack, which is a nightmare to set up on a laptop.

An open-source project called Colibri (Hummingbird) tries a different approach. Written in pure C with zero dependencies, it has earned roughly 36,000 GitHub stars, covers 9 model families ranging from 744B to 2.8T parameters, and claims frontier models can run starting from just 16GB of RAM.

Core Idea: Weights Are Data to Be Scheduled, Not State to Be Resident

Colibri's key insight is this: MoE models aren't too large to fit — they're just placed in the wrong location. Its design treats weights as data to be loaded on demand, rather than state that must permanently reside in VRAM.

Take the GLM series as an example: each token activates only about 40B parameters, roughly 5.4% of the total. Of that, the truly token-varying expert weights amount to about 11GB, while the dense routing components — attention, shared experts, embeddings — account for roughly 17B parameters (about 9.9GB after int4 quantization) and permanently live in memory without being shuffled around. The remaining 19,456 routed experts (each about 19MB, totaling ~370GB) sit on disk and are loaded on demand.

This mechanism is analogous to JIT compilation — a compiler doesn't compile an entire program at once; it compiles only the hot paths just in time. Colibri applies the same logic to a 744B parameter space: only when the router determines that a token needs a specific expert does it move that expert from NVMe to RAM to VRAM. Placement only affects speed — it doesn't change precision or routing semantics. That is the central promise of the entire design.

MoE (Mixture of Experts) is a sparsely activated neural network architecture: the model consists of many independent "expert" sub-networks, and for each token, a lightweight router selects only a few experts to participate in computation while the remaining expert weights sit completely idle. This allows the model's "total parameter count" and "active parameter count" to differ by an order of magnitude — GLM's 744B total but only ~40B activated per token is a direct result of this property. Traditional dense models (like early GPT series) pass every token through all parameters, requiring the entire weight matrix to reside in VRAM. MoE's sparsity naturally enables on-demand loading: since the vast majority of experts are never called upon for a given token, there is no reason to preload them all into VRAM. Colibri's three-tier hierarchy is built entirely on this premise.

Three-Tier Storage: VRAM, RAM, and Disk Each Play Their Role

Colibri treats VRAM, RAM, and disk as a unified inference scheduling hierarchy:

• VRAM: Holds hot experts. A GPU accelerates things but isn't a hard requirement — having one makes it faster, not possible.
• RAM: Holds the dense backbone plus learned hot experts. The longer you use it, the higher the hit rate — it tracks expert usage (Cola Usage) so predictions improve over time.
• NVMe: Cold experts rest on disk. A laptop with 25GB of RAM streaming from disk still produces correct outputs.

To hide disk latency, the engine performs one-layer-ahead prefetching (routing lookahead), achieving a measured predictive hit rate of 71.6% — disk wait time is masked by computation. It also supports dual-drive acceleration: a second SSD holds a read-only mirror, with load split by bandwidth weight (e.g., 9GB/s + 3GB/s), yielding roughly 33% speedup overall, with automatic fallback if the mirror is corrupted.

On the backend, CUDA, Metal, Vulkan, and NUMA share a single engine — AMD cards and older GPUs use Vulkan, Apple hardware uses Metal. The choice is based on your machine, not ideology.

NVMe (Non-Volatile Memory Express) is the mainstream high-speed SSD interface protocol. Compared to legacy SATA, NVMe connects directly to the CPU via PCIe lanes, with sequential read bandwidth typically ranging from 3–14 GB/s. This figure determines the speed ceiling for disk-streaming inference: the 25GB-RAM dev machine's measured 0.05–0.1 token/s is fundamentally disk-bound. One-layer-ahead prefetching (routing lookahead) is designed to overlap disk I/O with GPU/CPU computation — while layer N is being computed, the engine asynchronously fetches the experts needed for layer N+1 based on routing predictions, removing disk wait from the critical path. A 71.6% hit rate means that in most cases the data is ready when computation finishes; only on mispredictions does the engine actually have to wait on disk.

One Engine, Nine Model Families

The engine is small, but what it can feed is not. Nine families share one frontend, each with its own C file, accessed uniformly through coli chat, coli serve, and coli web. China's major model weights can all be run locally:

• GLM 5.2 / 5.3: 744B MoE, starts at 16GB
• GLM 5.3 Flash: 321B, 40B active, with vision, 25GB
• Kimi K3: 2.8T, starts at 32GB, native MXFP4 streaming from Moonshot
• DeepSeek^Ad V4 Flash: 284B, starts at 16GB, streams directly from the official repo, no conversion needed
• DeepSeek V4.1 Flash: 552B with vision, added as the 9th engine in V1.1.0
• Qwen3: 35B active, 3B; benchmarked at 1.44–10.05 token/s on two 8GB GPUs — roughly a 7× gap
• Plus multimodal models with audio and vision support

Switching machines only changes where the experts live — the model itself stays the same.

Real-World Benchmarks: Hardware Changes the Speed, Not the Answer

Community-reported benchmarks span the full range from workstations to laptops:

Hardware	Expert Location	Speed
6× RTX 5090	All in VRAM/RAM	9.0–9.2 token/s
Graviton 4, 64-core + 512GB	98% pinned in RAM	8.0 token/s
Dual Xeon + 1TB RAM	All pinned in RAM	5.42 token/s
MacBook Pro M5 Max 128GB	Metal unified memory	2.0 token/s
25GB dev machine	Streaming from NVMe	0.05–0.1 token/s

The 25GB dev machine is exactly the scenario the project was built for — slow is acceptable, but wrong is not. To reduce conversational cost, the engine also uses MLA to compress the KV cache: each token requires only 576 numbers instead of 32,768. Conversations can warm up, and restarts don't require re-prefilling from scratch.

MLA (Multi-head Latent Attention) is a KV cache compression technique proposed by the DeepSeek team and adopted by models including GLM. Standard multi-head attention requires storing full Key and Value vectors for each token (32,768 floats in GLM); as conversations grow, the KV cache rapidly balloons and exhausts limited memory. MLA projects KV into a low-dimensional latent space for storage and decompresses during computation, reducing per-token cache from 32,768 to 576 numbers — a ~98% reduction. For local inference where memory is already scarce, this means supporting longer conversation contexts without memory overflow, and being able to reload existing cache after a restart instead of reprefilling from zero every time.

Visual Workbench: Watch the Experts Fire

Colibri's coli web dashboard is not a diagram — it's a live workbench. It displays per-round latency and real-time VRAM/RAM/disk read activity. In the corner is a mini brain view called Atlas, which renders all 19,456 experts as a living cortex: color indicates storage tier, brightness indicates routing heat, and the experts activated in the current round flash white.

Experts analyzed by Atlas cluster by topic — poetry, law, Chinese, SQL, and so on. These positions come from measured routing distributions, not manual annotation. The embedding is draggable and rotatable. Watching a screen full of experts light up one by one as you type your prompt is genuinely compelling.

V1.1.0 Improvements and Hard Guarantees

V1.1.0 adds the DeepSeek V4.1 Flash engine (FP8 dense + FP4 experts; its 203GB N-gram memory reads only a few hundred bytes per pass, cutting cold-start time from 78.7 seconds to 25.1 seconds per round). It also fixes a cross-platform memory budget issue: previously, Windows and macOS would read Linux's memory files, causing the budget to collapse to 1GB and coli tune to immediately run out of memory. Now each OS correctly reads its own available memory.

The project makes several explicit guarantees: the default strategy will never silently alter progress or routing; when fast memory is insufficient, it only slows down; speculative decoding must be proven worthwhile before being enabled, otherwise it stays off. In the author's words — faithful, no bait-and-switch on the model.

Getting Started Is Actually Simple

The binary is only a few hundred KB, and the model comes with conversion utilities. Step one: get the engine — pre-built packages are available for Linux, macOS, and Windows; you can also build from source without a compiler. Step two: get a model — for example, the pre-converted int4 GLM 5.2 container is about 372GB. Common commands include coli chat (conversation), coli web (dashboard), coli serve (headless server), coli plan (view placement), coli doctor (readiness check), and coli tune (optimize for your machine). On Windows, just double-click coli.cmd to get started.

Three Sentences to Sum It Up

This project can ultimately be distilled into three sentences:

1. Models don't need to fit in VRAM — they need to be correctly placed.
2. Weights are data, not state that must be resident.
3. It gets faster the more you use it, because the engine is learning your workload.

Colibri, written in pure C with zero dependencies, unifies VRAM, RAM, and disk into a single inference hierarchy — 25GB of RAM and a patient SSD is all it takes to run a 744B model. For developers who want local control over their weights but are constrained by consumer hardware, this is a new path worth watching.