Bonsai

Providing small models with too many prompt tokens (tens of thousands to hundreds of thousands of specifications and tools) disperses their limited attention, and lightweight frameworks such as DSH are more small-model-friendly.

In the author's own testing, Bonsai processed a Zhihu-style question in 16 seconds per turn at an average of 235 tokens/s.

In concise mode, Bonsai is more prone to falling into 'Thunder Deep Think' due to a lack of available tools; in standard mode with ample tools, it more easily converges to calling a tool or producing the main text output.

Combining the Ninfer inference engine, Bonsai compressed model, and KVMem VRAM offloading, peak decoding reached 748 tokens/s and prefill peaked at 3.8K tokens/s.

Looking at each component individually, the performance gain is only about 10%–15%, but combining all three produces a notable synergistic effect (nonlinear acceleration).

A 'Thunder Deep Think' anomaly occurred: after the chain of thought completed, the model output 'stop' and halted without entering the main text generation phase; the author mitigated this by patching in detection of 'stop' followed by a forced jump to the main text branch.

When the context was set to 180K, compression should only be triggered at 80%–90% capacity, but in practice abnormal compression triggered at 20%–30%, causing reasoning results to be lost and creating a dead loop.

Bonsai compresses a 50-plus GB model down to 5–7GB while retaining considerable logical reasoning, coding, and multi-turn dialogue capabilities.

In the summarization task, 4 out of the 8 models failed: the 4-billion-parameter Qwen3.5 fabricated content, LFM2.5 altered facts, Bonsai added nonexistent information, and MiniCPM5 missed key points.

Bonsai uses 1-bit quantization, representing each weight with just one bit, making the entire model file only slightly over 1GB.