Xtier Plugin + Official SGLang: Hands-On with Hybrid VRAM/RAM Inference for MoE Models

Xtier Plugin + Official SGLang: Hands-On with Hybrid VRAM/RAM Inference for MoE Models

Bilibili creator "Yangzai" has released an open-source plugin called Xtier. Built on top of the official SGLang framework, it adds hybrid VRAM/RAM inference for MoE models without modifying the source code. Combined with official SGLang and optional acceleration packs, users can run large models that previously required professional servers on consumer-grade GPUs. Based on the video, this article outlines Xtier's design philosophy and real-world performance, with a focus on why MiniMax M3.1 Flash "falls flat."

What Is Xtier: A "Hybrid Inference" Shell for SGLang

According to the creator, Xtier is a plugin developed specifically for SGLang. The official SGLang team focuses on pure GPU inference, while Xtier adds CPU memory inference support on top, allowing MoE models to keep inactive expert parameters in RAM and load only the currently needed experts into VRAM.

"We build on their foundation—since they're all pure GPU inference, we add a CPU memory inference plugin on top. We rely on the official SGLang tree for shade."

The key to this design is "no source code changes, model-agnostic": any MoE model should theoretically work. The creator notes that the official framework actually has a KTransformers interface supporting MoE, but it's too inefficient and research-oriented. Xtier's goal is to raise the production readiness of this path.

Installation and Startup Process

Usage is simple: install official SGLang first, then install the Xtier plugin. On startup, the plugin prints detailed logs, going through stages of initialization, copying weights to VRAM/RAM, and loading experts. The creator says startup takes about three minutes on his device.

MoE (Mixture of Experts) models are the premise for understanding Xtier's value. Unlike traditional dense models that activate all parameters on every inference, MoE models consist of multiple "expert" subnetworks plus a routing gate, with each token activating only a few experts. This means total parameters can reach hundreds of billions, while per-inference activated parameters are far smaller. Xtier exploits this sparsity: expert parameters not currently selected by routing stay in CPU memory, and only active experts are moved into VRAM, allowing consumer GPUs with limited VRAM to host very large models. The cost is that expert switching requires data transfer over the PCIe bus—when activated experts are too large or memory bandwidth is insufficient, the GPU idles waiting for data, creating an inference bottleneck.

SGLang is one of the mainstream open-source LLM inference frameworks today, known for RadixAttention prefix caching and high-throughput scheduling, with the official version primarily targeting pure GPU deployment. It provides plugin and backend extension entry points, and Xtier hooks into these—so CPU memory inference capability can be layered on without forking or modifying SGLang source code, and it stays compatible with official releases as they iterate. The benefit of this "parasitic" development is low maintenance cost and universal optimizations that benefit all models; the limitation is being constrained by the granularity exposed by official interfaces, making deep customization of certain low-level scheduling strategies difficult.

Acceleration Pack Mechanism: Optional Speed vs. Precision Trade-Off

Another core design of Xtier is the "acceleration pack." Official SGLang plus the Xtier plugin will run, but speed may be only 65 tokens/s; adding a model-specific optimized acceleration pack can boost it to 137 tokens/s.

Acceleration packs are optional components, and the creator emphasizes that not all packs sacrifice precision. For example, the DeepSeek^Ad acceleration pack loses no precision at all; some models' packs incur a slight loss, such as 0.3%, in exchange for a boost from 76 to 90 tokens/s. Without an acceleration pack, inference is fully full-precision.

The creator also plans a "planner" feature, not yet complete. The planner will automatically recommend suitable settings and acceleration packs based on hardware parameters such as CPU compute, memory speed, VRAM compute, VRAM speed, and PCIe transfer speed, lowering the barrier for newcomers.

Development Journey: From Learning LS Jilang to Rebuilding the Engine

The creator candidly admits he initially learned from LS Jilang and LKMo, developed by Bilibili creator "Ai Tiaosheng de Nailong." LS Jilang is used for hybrid inference; the creator made some modifications and optimizations on top and validated them on an RTX Pro 5000 (50-series GPU).

But further optimization hit a wall: LKMo is closed-source, so the source code wasn't available—he could only "disguise as his LKMo" to optimize, and the work was mainly at the CPU scheduling level. The creator believes that to leverage 50-series GPU capabilities, the GPU portion must be touched, and the closed-source environment made this laborious.

So the creator chose to "break it apart and rebuild," developing Xtier directly on the official SGLang distribution, going through the official plugin entry point, and following official updates weekly. He explicitly states this is not cracking Longge's code—just referencing the ideas to build GPU-specific optimizations. The benefit is that universal optimizations can be developed once and benefit all models, while model-specific optimizations are packaged as acceleration packs, greatly reducing maintenance costs.

SM120 is the compute capability version for NVIDIA Blackwell architecture consumer GPUs (such as the RTX 50 series), while data center professional cards correspond to SM100. Different SM versions differ in instruction sets, tensor core features, and memory types; inference framework operator kernels need to be compiled and tuned for specific SM versions, otherwise they may fail to run or suffer significant performance degradation. MiniMax officially only adapted to SM100 professional servers; after the creator added SM120 support, speed improved from 45 tokens/s to 51.6 tokens/s. This also shows that much of the open-source community's work adapting to consumer GPUs involves SM version compatibility at the operator level.

MiniMax M3.1 Flash Hands-On: Slow Speed, Frontend "Totally Botched"

The focus of this video is testing MiniMax M3.1 Flash. The creator's assessment is quite direct: "Honestly, it's a total letdown—genuinely garbage."

Why It's Slow

MiniMax M3.1 Flash has 426B native parameters, and the official release is a quantized version with model files still over 200GB. More critically, each token requires activating 23.5B parameters, corresponding to about 13.2GB of activated expert data. By comparison, GLM 5.3's activated experts total only 8.8GB.

The creator made a fix for SM120 devices (officially only SM100 professional servers were supported), improving speed from 45 tokens/s to 51.6 tokens/s, but it's still slow. The reason is that CPU and memory speed can't keep up with the GPU: the GPU finishes computing and waits for the CPU, wasting a lot of time. The yellow bars in the video show the proportion of wasted time—MiniMax wastes more than half, while Qwen 3.8 wastes almost none.

"The GPU is too strong, the CPU isn't strong enough, compute is too weak, memory speed is too slow. The GPU finishes and waits for the CPU to compute, so it wastes time."

Activated parameter count is a key metric for MoE inference cost, explaining actual speed better than total parameter count. MiniMax M3.1 Flash activates about 23.5B parameters per token, corresponding to 13.2GB of activated expert data, while GLM 5.3 has only 8.8GB. In a hybrid inference architecture, this data must be transferred from memory to VRAM over PCIe for every token generated—the larger the data, the longer the transfer. When GPU compute far exceeds CPU and memory bandwidth, the GPU can only wait for the next batch of expert data after finishing computation; this idle time is the "waste" shown by the yellow bars in the video. Therefore, the bottleneck in hybrid inference is often not the GPU, but the match between memory bandwidth and PCIe transfer speed.

Task Test: Frontend Ability Worse Than Previous Generation

The creator gave MiniMax a task: write a model launcher from scratch, referencing a version DeepSeek had written earlier (described as "very sci-fi and user-friendly"), but with the original launcher code removed, requiring an independent implementation.

The interface MiniMax produced, while fully functional—able to select models like Qwen, MiniMax, DeepSeek, and GLM, with a model path input field and a run button—was crudely designed, requiring users to manually fill in paths. The creator rated its frontend level as "about the same as the previous generation Qwen 3.6 27B," with poor leap capability.

Speed Comparison and Current Status

The creator gave current speed performance for various models on Xtier (based on his RTX Pro 5000 platform):

• Qwen 3.8: fastest, essentially no waste
• DeepSeek 5.3: on par with the modified LS Jilang, slightly slower by 2%; the creator is still investigating why
• GLM 5.3: faster than LS Jilang
• MiniMax M3.1 Flash: very slow

The creator says the deeper reasons for MiniMax's slowness will be explained in the next video. The Xtier project is fully open source, allowing users to modify and adapt to different GPUs (40-series, 30-series, etc.) and to create their own acceleration packs.

For users wanting to run niche MoE models, Xtier offers a low-barrier path: get it running first, then gradually optimize speed through community or self-made acceleration packs.

One-line Summary

Open-source plugin Xtier enables hybrid VRAM/RAM inference for MoE models on consumer GPUs; MiniMax M3.1 Flash performs poorly in hands-on tests.

Paragraph Summary

Bilibili creator "Yangzai" released the open-source plugin Xtier, built on the official SGLang framework without source code changes, adding CPU memory inference for MoE models—keeping inactive experts in RAM and loading active experts into VRAM, enabling consumer GPUs to run large models previously requiring professional servers. Xtier offers optional acceleration packs: DeepSeek's pack boosts speed losslessly, while some models trade about 0.3% precision for speed. A planner feature is still in development. In testing, Qwen 3.8 was fastest, DeepSeek and GLM performed normally, while MiniMax M3.1 Flash—activating 23.5B parameters per token and 13.2GB of active experts—was bottlenecked by CPU and memory bandwidth lagging behind the GPU, causing extensive idle waiting, reaching only 51.6 tokens/s, and its frontend generation ability was rated worse than the previous generation Qwen 3.6 27B.