Stratavie
An open-source local inference engine released under the MIT license, designed specifically for running large-scale Mixture-of-Experts (MoE) models locally. It supports multi-tier hardware collaboration across GPU VRAM, system memory, and SSD storage, enabling execution of very large parameter models on lower-specification hardware configurations.
Timeline (last 90 days)
Stratavie 0.1.41 is an open-source inference engine under the MIT license, specifically designed for local execution of ultra-large MoE models, with 18,400 stars and 216 contributors on GitHub.
This solution uses a 4-billion-parameter small model as the draft model, which comes from the same series as the 125-billion main model, achieving a 1.6× to 1.8× speedup.
A third-party same-VRAM comparison on Zhihu shows that Stratavie writes responses at 62.9 tokens/second, while Llama.cpp achieves 23.4 tokens/second, with long-document reading being nearly 10 times faster.
The runtime uses a four-tier storage architecture: the GPU maintains approximately 3.5 GB for attention/routers/shared experts and caches approximately 4,500 high-frequency experts; RAM stores all 24,576 experts; and the SSD stores approximately 28.8 GB of parameter lookup tables and N-Gram group tables.
Stratavie processes only one request at a time by default, with multiple requests queued; enabling parallel mode on a 12 GB GPU will slow down each individual request.
Stratavie provides an OpenAI-compatible interface (/v1) and an Anthropic interface (/v1/messages), with hardware support covering NVIDIA RTX 20 through 50 series GPUs and several AMD graphics cards.
All Facts (6)
Stratavie 0.1.41 is an open-source inference engine under the MIT license, specifically designed for local execution of ultra-large MoE models, with 18,400 stars and 216 contributors on GitHub.
50%UnverifiedThis solution uses a 4-billion-parameter small model as the draft model, which comes from the same series as the 125-billion main model, achieving a 1.6× to 1.8× speedup.
50%UnverifiedA third-party same-VRAM comparison on Zhihu shows that Stratavie writes responses at 62.9 tokens/second, while Llama.cpp achieves 23.4 tokens/second, with long-document reading being nearly 10 times faster.
50%UnverifiedThe runtime uses a four-tier storage architecture: the GPU maintains approximately 3.5 GB for attention/routers/shared experts and caches approximately 4,500 high-frequency experts; RAM stores all 24,576 experts; and the SSD stores approximately 28.8 GB of parameter lookup tables and N-Gram group tables.
50%UnverifiedStratavie processes only one request at a time by default, with multiple requests queued; enabling parallel mode on a 12 GB GPU will slow down each individual request.
50%UnverifiedStratavie provides an OpenAI-compatible interface (/v1) and an Anthropic interface (/v1/messages), with hardware support covering NVIDIA RTX 20 through 50 series GPUs and several AMD graphics cards.
50%