Colibri
An open-source large model inference engine written in pure C with zero external dependencies. It implements a three-tier storage hierarchy (VRAM / RAM / NVMe) combined with an on-demand expert-loading mechanism, enabling Mixture-of-Experts models with very large parameter counts to run on consumer-grade hardware.
Core Facts
Timeline (last 90 days)
The ceiling on inference speed when using the hard disk as the backend is constrained by disk I/O, and the actual experience in long-text and high-concurrency scenarios is limited.
Colibri officially provides pre-compiled versions for three platforms; users can enter a conversational state by downloading the engine and model and then executing three commands.
Colibri's official stance makes no promises regarding inference speed, but provides a hard guarantee on semantics (output correctness and consistency with the model's semantics).
Colibri is an open-source inference engine that uses the hard disk as extended memory, reading model shards on demand, enabling ordinary computers to run ultra-large-scale models (such as 2.8T parameters).
Colibri adopts an on-demand expert loading approach, loading the corresponding model shards from disk only when needed, a strategy that is especially well-suited to the MoE (Mixture of Experts) architecture.
Colibri supports model parameter scales ranging from 744B to 2.8T.
Colibri supports nine major model families, including popular open-source models from both domestic and international sources such as GLM, Kimi, and DeepSeek.
Colibri reached third place on the open-source weekly rankings and has accumulated approximately 37,000 stars.
Colibri's backend supports CUDA, Metal, Vulkan, and NUMA all sharing a single engine.
Colibri provides unified access via coli chat, coli serve, and coli web, supporting models including GLM 5.2/5.3, Kimi K3, DeepSeek V4 Flash, and Qwen3.
15 more timeline events
All Facts (20)
Open-source models publicly release their complete weights, allowing anyone to download, modify, run locally, or even develop commercially.
65%UnverifiedThe ceiling on inference speed when using the hard disk as the backend is constrained by disk I/O, and the actual experience in long-text and high-concurrency scenarios is limited.
50%UnverifiedColibri is an open-source inference engine that uses the hard disk as extended memory, reading model shards on demand, enabling ordinary computers to run ultra-large-scale models (such as 2.8T parameters).
50%UnverifiedColibri adopts an on-demand expert loading approach, loading the corresponding model shards from disk only when needed, a strategy that is especially well-suited to the MoE (Mixture of Experts) architecture.
50%UnverifiedColibri supports model parameter scales ranging from 744B to 2.8T.
50%UnverifiedColibri supports nine major model families, including popular open-source models from both domestic and international sources such as GLM, Kimi, and DeepSeek.
50%UnverifiedColibri officially provides pre-compiled versions for three platforms; users can enter a conversational state by downloading the engine and model and then executing three commands.
50%UnverifiedColibri's official stance makes no promises regarding inference speed, but provides a hard guarantee on semantics (output correctness and consistency with the model's semantics).
50%UnverifiedColibri's one-layer-ahead prefetching (routing lookahead) achieves a measured prediction hit rate of 71.6%.
50%UnverifiedColibri's backend supports CUDA, Metal, Vulkan, and NUMA all sharing a single engine.
50%UnverifiedColibri provides unified access via coli chat, coli serve, and coli web, supporting models including GLM 5.2/5.3, Kimi K3, DeepSeek V4 Flash, and Qwen3.
50%UnverifiedOn 6 RTX 5090 GPUs, Colibri's measured speed reaches 9.0–9.2 tokens/s, while on a 25GB development machine streaming from NVMe the speed is 0.05–0.1 tokens/s.
50%UnverifiedColibri covers 9 model families with parameters ranging from 744B to 2.8T, claiming that frontier models can run starting from 16GB of memory.
50%UnverifiedColibri adopts a three-tier storage hierarchy design of VRAM, RAM, and disk, placing hot experts in VRAM, dense backbone layers and hot experts in RAM, and cold experts on NVMe disk for on-demand loading.
50%UnverifiedWeight placement of MoE models only determines speed and does not alter precision or routing semantics.
50%UnverifiedColibri supports dual-disk acceleration with bandwidth-weighted traffic splitting, yielding an overall speed improvement of approximately 33%.
50%UnverifiedColibri includes a coli web visualization dashboard featuring an Atlas brain view that visualizes 19,456 experts, color-coded by storage tier and with brightness indicating routing heat.
50%UnverifiedColibri makes the following commitments: the default policy will not silently alter progress or routing, speed will only be reduced rather than accuracy when high-speed memory is insufficient, and speculative decoding must be proven worthwhile or it will be disabled.
50%UnverifiedColibri is an open-source large model inference engine written in pure C with zero dependencies.
50%UnverifiedColibri is written in pure C with zero runtime dependencies and introduces a three-tier memory model: hot experts reside in VRAM, warm experts reside in system memory, and cold experts are streamed from NVMe SSD, enabling loading of very large models like GLM within 25GB of memory.
50%