Colibri Open-Source Engine: Run 2.8T LLMs on a Regular PC

A New Approach to Running Large Models on Consumer Hardware

Running a model with 2.8 trillion parameters has traditionally been the exclusive domain of data centers — requiring racks of high-end GPUs and budgets that can easily exceed tens of thousands of dollars. An open-source inference engine called Colibri is trying to change that. As covered by tech creators on Bilibili, the engine recently ranked third on the open-source weekly chart and has accumulated around 37,000 GitHub stars. Its core pitch is simple: let ordinary computers load and run massive-scale models.

Colibri's approach is straightforward but pragmatic. Rather than forcing the entire model into VRAM or RAM, it uses your hard drive as extended memory, loading only the relevant model shards from disk on demand — a strategy called "expert-on-demand loading." For large models built on the MoE (Mixture of Experts) architecture, this kind of on-demand scheduling is a natural fit: only a small fraction of the total parameters actually get activated during any given inference pass.

Coverage: Nine Model Families from 744B to 2.8T

In terms of parameter scale, Colibri covers an impressive range — from 744B all the way up to 2.8T. This means it wasn't built for a single model, but rather designed to be compatible with a broad ecosystem of mainstream large models.

According to the video coverage, it supports nine model families, including GLM, Kimi, and DeepSeek^Ad (referred to as "DeepSync" in the video, which appears to be a misspeak), among other popular domestic and international open-source models. For everyday users, the practical value is clear: no need to set up a separate runtime environment for each model — one engine handles them all on the same hardware.

Why the "Disk Instead of VRAM" Strategy Actually Works

At its core, this is a bandwidth-for-cost tradeoff. VRAM and RAM are fast but expensive; hard drives — especially NVMe SSDs — offer large capacity at a fraction of the cost. For MoE models with sparse activation patterns, storing rarely-used expert weights on disk and reading them only when needed can dramatically lower the hardware barrier to entry, at the cost of some speed. This also explains why the developers make no promises about speed — the real question they're solving is whether you can run these models at all, not how fast they'll run.

The MoE (Mixture of Experts) architecture is key to understanding why Colibri works the way it does. In a traditional dense model, every token passes through all parameters. In a MoE model, each transformer layer contains multiple "expert" sub-networks, and a lightweight "router" decides which few experts (typically 2) each token gets sent to. This allows the total parameter count to be enormous (e.g., 2.8T), while the parameters actually activated during a single inference pass remain relatively small (e.g., DeepSeek-V3 activates roughly 37B parameters per pass). Popular open-source models like DeepSeek and Kimi both use MoE or MoE-variant architectures — which is precisely what makes the "nine model families" pitch technically viable. Their sparse activation patterns are what make on-demand expert loading from disk feasible in practice.

Getting Started: Three Commands to Your First Chat

For everyday users, one of the most appealing aspects is how streamlined the setup process is. Colibri offers pre-compiled binaries for three platforms, so users can download the engine, download the model, and run three commands to start chatting — no complex compilation from source or dependency wrangling required.

Worth noting is how the developers frame the tool's limitations: no promises on speed, but a hard guarantee on semantics. In other words, the priority is ensuring output correctness and semantic consistency with the model — actual speed will vary depending on local disk read/write performance, model size, and other factors. This kind of honest positioning is more trustworthy than inflated performance claims.

Open-Source License and How to Get It

Colibri is released under the Apache 2.0 license, a permissive open-source license that is quite friendly for commercial use, allowing developers to build on top of it or integrate it into other projects.

The video also highlights an important reminder: model weights must be downloaded from each model's official channels. This is a compliance point when using open-source LLMs — avoid downloading weights from unofficial or unknown sources. Resources for the tool have been compiled into free resource collections like "开工AI," with Chinese-language documentation to help lower the barrier for new users.

A Measured Take

Colibri's significance lies more in accessibility than raw performance. It transforms "impossible" into "possible" when it comes to running massive models locally — a genuine win for researchers, enthusiasts, and budget-constrained developers. But it's worth being clear-eyed: disk-backed inference has a speed ceiling set by disk I/O. Real-world performance in long-context or high-concurrency scenarios will need to be tested case by case.

For those who want to experiment, it's an excellent low-cost entry point to frontier-scale models. For production workloads demanding low latency, dedicated hardware remains irreplaceable. What this tool really does is give the choice back to the user.

Background Notes

NVMe SSD sequential read speeds typically fall in the 3,000–7,000 MB/s range (PCIe 4.0/5.0), far below GPU VRAM bandwidth (e.g., H100's 3.35 TB/s), but already tens of times faster than mechanical hard drives. One of the key engineering challenges for engines like Colibri is using prefetching and async I/O to hide disk latency — loading the next likely batch of expert weights from disk into memory while the GPU/CPU is still processing the current expert layer, forming a pipeline that reduces wait time. The sparse activation characteristics of MoE models (typically only 2–8 experts activated per inference, out of potentially dozens or hundreds) pair naturally with this scheduling strategy: the sparser the activation path, the less data needs to be fetched from disk per inference, keeping speed degradation more manageable. This is also why the same "disk-instead-of-VRAM" approach works much less well for dense models — dense models require reading all parameters at every layer, multiplying I/O pressure significantly.