748 token/s on a Single Local GPU: Pushing the Limits with Bonsai2 + Ninfer + KVMem

Three Powerful Tools, Combined at Once

Before the National Day holiday, a Bilibili creator — building on technical documentation and deployment plans shared by the well-known community figure "Shen Sanshu" — managed to run a compressed large language model on a local setup with just 16GB of VRAM. At the time, decode speed was approaching 400 token/s and prefill hit around 2,500 token/s, which was already impressive. During the holiday, the upstream project continued iterating rapidly. When the author returned in early October and picked things back up, they found that the new version had done something critical: it integrated Ninfer inference engine, Bonsai compressed model, and KVMem VRAM offloading all together.

The result wasn't a simple linear improvement — it was an unexpected performance leap. Peak decode speed hit 748 token/s, while prefill (what the author calls "prefill/prefill rush") reached up to 3.8K token/s. Compared to the previous ~400, decode speed nearly doubled. The author admits: "Even now I don't know how this number came about," because looking at each component individually, the gains were only around 10–15%. Yet their combination produced a clear synergistic effect.

Where the Speed Doubling Comes From: The Chemistry of Speculative Decoding

The core of this performance leap lies in the fusion of multiple prediction heads at the decode layer. The previous setup used MTP and D-Flash prediction heads — D-Flash offered higher hit-rate gains when VRAM was abundant, which is how the original ~400 token/s was achieved. The new version mixes D-Flash (2nd gen), MTP, and N-gram speculative decoding together, and something changed.

The author is emphatic about this: putting D-Flash2 and N-gram together is not a simple addition or multiplicative boost — it produces a doubling-level effect. The essence of speculative decoding is to cheaply predict multiple tokens, then have the main model verify them in batch. The higher the hit rate, the more valid tokens produced per verification step. With multiple prediction mechanisms complementing each other in the same pipeline, the overall hit rate is elevated — which is why single-concurrency speed shot into the 700s.

It's worth emphasizing: this is a single-request (single-concurrency) result. The author repeatedly cautions: don't compare this to multi-concurrency server throughput. Ninfer, an engine designed specifically for Qwen-class models, inherently runs about twice as fast as general-purpose solutions like llama.cpp. Hitting 740+ at single concurrency is already remarkable.

Speculative Decoding is a technique for accelerating large language model inference. The core idea is "let a small model guess, let the big model verify." A lightweight draft model or prediction head quickly generates several candidate tokens, which are then handed to the main model for parallel batch verification — tokens consistent with the main model's distribution are accepted; others are rejected. Since verifying multiple tokens in parallel is far cheaper than generating them one by one autoregressively, overall throughput increases significantly. Hit rate is the key metric: the higher the hit rate, the more valid tokens produced per verification round, and the greater the speedup. MTP (Multi-Token Prediction), D-Flash, and N-gram are prediction mechanisms from different sources and at different granularities — N-gram leverages local repetition patterns in already-generated text for cheap predictions, while MTP and D-Flash are multi-head or auxiliary decoding structures at the model level. Stacking multiple mechanisms means there's always at least one with a high hit rate for any given context, which collectively raises the average hit rate — and this is the fundamental reason why the combination produces nonlinear acceleration.

Deployment Flexibility from KVMem

The other major development is KVMem. Its role is to offload the KV cache pool from VRAM to system memory, freeing up precious GPU memory. For local users stuck with 16GB VRAM, this means being able to try larger quantized models without hitting a VRAM wall.

One previous pain point was that KVMem and Ninfer were originally incompatible — the two had conflicts in encoding and cache handling. The author suspects the upstream team either coordinated directly with the KVMem author or resolved the compatibility layer themselves. Either way, the two now work simultaneously. This opens up enormous possibilities for deployment: the author is currently experimenting with loading a standard Q2/Q3 quantized 27B model into this environment, pairing mixed speculative decoding with Ninfer's high-speed engine, to see if solid speed is still achievable even with KV stored in system memory.

For reference: previously, using llama.cpp with KVMem for a normal model could reach around 100+ token/s. Moving KV to system memory was expected to be slow — but combined with speculative decoding, that assumption may not hold, which is exactly the direction worth exploring further.

KV Cache is the data structure used by Transformer models during autoregressive inference to store intermediate results from the attention mechanism. Each time a new token is generated, the model needs to access the Key and Value vectors for all previous tokens. Caching these in VRAM avoids recomputation, but at the cost of memory usage that grows linearly with context length. In long-context scenarios (e.g., 32K, 180K), the KV cache often consumes more memory than the model weights themselves, making it the primary VRAM bottleneck for local deployment. KVMem addresses this by offloading this data to system DRAM — trading bandwidth for capacity. DRAM bandwidth is lower than VRAM, which theoretically introduces latency. However, combined with speculative decoding, where each verification round produces more tokens, the per-access time cost is amortized, making the actual speed penalty far smaller than expected.

Real-World Pain Points: "Thunder Deep Think" and Abnormal Compression

Deploying this stack was far from smooth. The author progressively documented the pitfalls encountered. The most notable issues involve what they call "Thunder Deep Think" (extended reasoning loop) anomalies:

• Outputs stop but never generates the response body: After the chain-of-thought completes, the model neither hits the thinking budget limit nor runs out of VRAM — it simply outputs stop and halts without entering the response generation phase. The author previously wrote a patch: once stop is detected, the engine is forced to jump to the response branch and continue generating, without breaking the existing logic. Upstream later released a more elegant frontend-side fix; after multiple rounds of retesting, the author confirmed the issue was mitigated and migrated to the new upstream baseline (which also fixed other bugs), then recompiled.
• Inexplicably early context compression: With context set to 180K, compression should ideally trigger at 80–90% capacity. But in practice, it was triggering at 20–30%, causing reasoning progress to be lost, forcing repeated re-reasoning, and creating a feedback loop. The author suspects this abnormal early compression is one trigger for the loop, but the root cause remains unclear — they've put out an open call to the community for ideas.

Both issues stem partly from engine implementation, and may also be amplified by the interaction between the 27B base model's long-reasoning characteristics (which still can't resolve after 32K tokens of thought) and post-quantization degradation.

"Thunder Deep Think" refers to a state where a model with long chain-of-thought (CoT) capabilities enters an extremely long internal reasoning loop during inference. Models like the DeepSeek^Ad-R1 series are trained to engage in extensive internal reasoning before generating output — but after quantization compression, the ability to "know when to stop thinking" may degrade, causing the reasoning chain to extend indefinitely without ever entering the response phase. Context compression is an engineering technique for handling long contexts: when context length approaches the configured limit, the system automatically summarizes or trims historical content to free up space for continued generation. Ideally, compression should only trigger near capacity limits. But if the compression logic conflicts with the model's state detection for reasoning, it may fire mid-inference — cutting off intermediate reasoning results and forcing the model to start over, creating a "reasoning → compression → re-reasoning" death loop.

How to Fairly Evaluate Compressed Models Like Bonsai

Beyond the speed numbers, the author spent significant time discussing how to properly assess extremely compressed small models like Bonsai. They raised several "unfair" framings, with some genuinely thought-provoking conclusions:

First, the comparison baseline is wrong. Comparing it to MoE or Q2/Q3 27B models that can run locally, yes, it falls behind on quality. But compared to larger models from a year or two ago, it holds its own on many capabilities. Compressing a 50GB+ model down to 5–7GB while preserving solid logic, coding, and multi-turn dialogue is a remarkable engineering achievement in itself.

Second, its defining trait is "dumb-fast." The author tested a Zhihu-style question and got a 16-second response at an average of 235 token/s. For tasks like debugging code, it's basically fine. Speed itself is efficiency — the 740+ single-concurrency experience feels great in interactive settings.

Third, bigger isn't always better. Since the Agent paradigm took off earlier this year, the industry has increasingly recognized that capability, role definition, tool-use, and framework compatibility matter more than raw parameter count. Give a small model the right framework, skills, and environment, and its practical utility gets meaningfully amplified — a model that can only chat in a web interface can actually get work done inside frameworks like DSH or Code Buddy.

Fourth, extreme compression has long-term technical significance. Like video and audio compression, compressing large models to a size that fits on phones or edge devices has profound implications for AI accessibility and commercialization. The author is optimistic that in the future, 8GB GPUs paired with these quantization techniques could handle what today requires hundred-billion-parameter models.

Getting the Most Out of Small Models: Frameworks, Prompts, and Behavioral Intervention

The author observed clear behavioral patterns in Bonsai: it's lazy, wants to finish quickly, avoids proactive searching, and resists tool-calling. In minimal (no-tool) mode, it's more likely to spiral into "Thunder Deep Think" — because with nothing at hand, it has no choice but to keep reasoning. In standard mode with tools available, it's more likely to converge on either calling a tool or generating a response.

From this, they proposed two engineering-oriented approaches:

1. Give it a lightweight but clearly structured environment. DSH may be more small-model-friendly than traditional heavy frameworks, with lighter system prompts and constraints. By contrast, some frameworks dump tens of thousands to hundreds of thousands of tokens of prompts, rules, and tools on the model — easily overwhelming a small model's already limited attention.
2. Use prompts and skills for behavioral intervention. For example, explicitly telling it "past a certain point, you cannot keep reasoning — you must commit and start generating output, even if it's just three to five words." Or triggering web search via skill invocation — in practice, when explicitly asked to search, it actually does a decent job; it just won't do it proactively.

The author also mentioned having a comprehensive code workflow specification (which they call "SDV") — emphasizing a closed loop of problem analysis, decomposition, planning, evidence collection, review, testing, hallucination prevention, and rollback capability. Under this spec, output quality from different models tends to converge. Large models have internalized engineering planning and iterative discipline; small models haven't, which is why they need external frameworks to compensate for weaknesses and play to their strengths.

Summary

The value of this experiment isn't in any single number — it's in demonstrating the flexibility of the Ninfer + Bonsai + KVMem combination: speculative decoding fusion delivers 740+ token/s at single-card, single-concurrency on the extreme end; KVMem offloads the KV cache to system memory, freeing VRAM to accommodate larger quantized models; together, the stack gives local deployment significantly more room to maneuver. On the other side, issues like "Thunder Deep Think" and abnormal early context compression still await community effort to resolve. For those looking to reproduce this, the author recommends feeding the relevant technical documentation and build process directly to an AI, then following along step by step according to your own environment.