Unverified50% confidenceBenchmarkExact time
Combining the Ninfer inference engine, Bonsai compressed model, and KVMem VRAM offloading, peak decoding reached 748 tokens/s and prefill peaked at 3.8K tokens/s.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/10/2026
First Seen
Valid until: 1/8/2027
Sources
Related Entities
Related Claims
UnverifiedBonsai uses ternary quantization, compressing each weight to one of three values—-1, 0, or +1—with a shared scaling factor per 128 weights, requiring approximately 1.7 bits per weight, roughly 9 times smaller than 16-bit full precision.70% similarUnverifiedWhen running Qwen3-Next-Flash (Autoround W4A16 quantization), the prefill speed was approximately 1.3k tokens/s, and the generation speed reached 70 tokens/s (code) / 60 tokens/s (prose), with support for 128k+ long context.68% similarUnverifiedFor a typical 7B-parameter model, the KV Cache per 1K tokens requires approximately 0.5–1 GB of VRAM.67% similarUnverifiedDeveloper Primitive reported: 91.2 tokens/s with speculative decoding off, reaching 142.6 tokens/s after enabling 3 speculative tokens, approximately a 56% speedup under the same configuration.67% similarUnverifiedA 3B model typically achieves an inference speed of 10–30 tokens/second on a modern CPU via llama.cpp.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/995001API
curl https://kongchang.com/api/v1/knowledge/claims/995001MCP
get_claim(id=995001)