Running a 27B Model on 6GB VRAM? The Truth and Limits of Bonsai 2's Ternary Quantization

Prism ML's Bonsai 2 makes a tempting promise: retain 98% of a 27-billion-parameter model's performance in under 8GB of VRAM. In other words, a standard consumer GPU — or even a MacBook — could run a model that would normally require 54GB of memory. It sounds like a dream come true for local deployment. But does the intelligence actually carry over? One YouTube creator put Bonsai 2 and the original Qwen3-27B on the same track with their own benchmarks, and the results revealed a blind spot in compression technology that often goes overlooked.

What Bonsai 2 Actually Is: From Rounding to Training a New Format

To understand what makes Bonsai 2 special, you first need to understand how it differs from standard quantization. At full precision, each weight in Qwen3-27B is stored as a 16-bit float, putting the entire model at roughly 54GB — well beyond the memory limits of most laptops and consumer GPUs. The conventional approach is quantization, which is essentially "rounding": take an already-trained model and reduce the precision of each weight to store it in fewer bits. It's cheap and fast, and it's what most models on the market use.

But round aggressively enough, and the model starts losing intelligence — because intelligence lives in the weights. According to Prism ML's own comparison table, a standard 2-bit version weighs about 7.3GB but retains only around 88% of benchmark scores. A smarter approach is "selective quantization" — keeping the most critical parts at higher precision while aggressively compressing the rest — but it's still fundamentally rounding.

Bonsai 2 takes a completely different route. Starting from the pretrained Qwen3-27B, it is trained into a new format rather than simply rounded into one. Prism ML says the training was done on Google's TPUs, but hasn't disclosed the methodology, data, or compute used — that part remains a black box. What we have access to is only the final weights and the quantization format they use.

The Clever Trick: Ternary Quantization and Rotation Matrices

Bonsai uses ternary quantization, compressing each weight into one of three possible values: -1, 0, or +1. Large positive weights become +1, large negative weights become -1, and small ones are zeroed out. To preserve the original magnitude of the weights, a scaling factor is introduced: every 128 weights share a scale, with the real weight approximately equal to scale × ternary value. This means each weight takes only about 1.7 bits to store — roughly 9× smaller than 16-bit full precision.

There's also a key trick: in a standard neural network, a few very large weights in a group can completely skew the shared scale for all 128 weights. To address this, Prism ML applies a rotation matrix before converting to ternary values, smoothing out those spikes and distributing the energy across more dimensions, so extreme values are no longer as disruptive. This makes packing everything into three discrete values much cleaner. They claim this is precisely what helps preserve the model's intelligence, allowing Bonsai to score 98% on key benchmarks despite being so much smaller.

One important caveat to keep in mind: that 98% is an average across 20 benchmarks, measured with the model's "thinking" mode at maximum intensity. This condition will come back to matter in the actual tests.

---

The role of the rotation matrix deserves more explanation. Neural network weight distributions are typically uneven — a small number of "outliers" are far larger than the rest. In ternary quantization, if one weight in a group of 128 has an unusually large absolute value, the shared scale must accommodate that extreme, causing all other weights to be compressed too coarsely and concentrating information loss. A rotation matrix is essentially an orthogonal transformation: it doesn't change the length or angular relationships of vectors, but it redistributes energy from a few dimensions across many — similar to spreading a long shadow across multiple directions by rotating. After the rotation, the weight distribution becomes more uniform, outliers are diluted, and the 128 weights can share a more reasonable scale, preserving the accuracy of the ternary encoding. This idea draws from research like BitNet and QuIP#, with the core insight being: preprocessing before quantization often has a greater impact on accuracy than the quantization algorithm itself.

---

Simple Tasks: The 98% Promise Mostly Holds

The tester's setup was rigorous. On one side, Bonsai 2 running locally on a Mac via Prism ML's own llama.cpp fork; on the other, full-precision Qwen3-27B accessed through OpenRouter — the exact comparison Prism ML uses officially. Both models ran inside the same agentic coding framework, with identical tools, prompts, and instructions. To get a clean baseline of the model itself, thinking mode was disabled on both sides.

On simple tasks, the two were nearly neck and neck. For an SVG of an octopus playing drums, Qwen scored 7 and Bonsai scored 6. An analog clock that needed to display a specific time correctly was tough for both — Qwen got 3, Bonsai got 4. On a Breakout game debugging task with 6 planted bugs, Bonsai actually fixed 5 and scored 8.5, while Qwen scored 7.

Two short agentic tasks — a sales spreadsheet with hidden duplicate rows and mixed currencies, and a corrupted refund report in a database — were handled equally well by both models, catching every trap and completing the fixes.

Long-context tests were also encouraging: given a real codebase of roughly 100K tokens with planted facts and 7 questions, both models got everything right. In short, for tasks completable in one or two steps — even with thinking disabled — the 98% promise holds up reasonably well, with very similar patterns of success and failure.

Long-Chain Agentic Tasks: Bonsai Completely Falls Apart

The real dividing line appeared in the second round — six real-world long-chain build tasks: a black hole simulation with gravitational lensing, a 1-bit-style pirate ship battle, a voxel sea turtle on a coral reef, a real-time ISS tracker with a 3D Earth, a landing page generated from a research brief, and a full Linear clone with teams, cycles, keyboard shortcuts, and undo functionality. These tasks required writing multiple files, running them, checking screenshots, and fixing problems — often dozens or even hundreds of steps.

Qwen scored 35/60. Its Linear clone passed 19 of 24 requirements, the landing page passed 11 of 13, and the ISS tracker passed 9 of 13. It stumbled on the toughest black hole physics and the strict pixel rules of the pirate scene, but it at least delivered real, working applications.

Bonsai, on the other hand, completely collapsed across all six tasks — producing not a single working page, with five of the six failing to generate even a page in the project directory. The tester also pulled up earlier tests from ten days prior on a DGX Spark, where Bonsai had plenty of compute available yet failed in exactly the same way.

What the Logs Reveal: Going in Circles

The cause of failure is buried in the logs — and it's worth paying close attention to. On difficult tasks, Bonsai frequently got stuck in a "looking around" phase: on the ISS tracker task, it made 153 tool calls, 150 of which were just reading files, listing files, and searching. Another issue was infinite loops — it would run a tool, get a result, then run the same tool to get the same result, repeating indefinitely. In one case, it called the same tool 114 times in a row. By comparison, Qwen's worst repetition across all six projects was just 6 times.

Qwen wasn't without its own flaws — it spent about 40 minutes chasing a syntax error it had introduced in the pirate scene, and had a habit of rewriting entire files instead of making local edits. But those are the mistakes a model that's building something makes. Bonsai's pattern was fundamentally different: it could write clean, error-free code, but seemed unable to recognize that it had already done something, leaving it unable to make forward progress on hard tasks.

---

The "going in circles" behavior Bonsai exhibits corresponds to a concept in AI agent research: context amnesia or long-range coherence degradation. When generating each step, a large language model relies on its entire conversation history (the context window) to maintain an internal state of "what has already been done." When a model is heavily compressed, its ability to effectively process long contexts may degrade far faster than on short-sequence tasks — quantization errors accumulate layer by layer through each attention layer, and after dozens of steps, the model's "memory" of prior actions becomes severely distorted, manifesting as repeated execution of the same operation without recognizing the repetition. It's somewhat like short-term memory decline under human fatigue: individual actions can still be performed, but the ability to monitor oneself across time breaks down first. This also explains why short tasks are largely unaffected — when context is short enough, errors haven't yet accumulated enough to interfere with self-checking.

---

Why This Happens: The Truth Hidden Behind Averages

Interestingly, Prism ML's own whitepaper points to the same problem. That 98% figure comes from 20 shorter benchmarks, but the whitepaper also reports results on two long-chain agentic benchmarks — which were not included in the average. On TerminalBench 2.1, Bonsai retains only about 76% of the original model's score; on SWE-bench Verified, built from real GitHub issues, it's about 75%.

For a model of this size, 75% is still a respectable result — but it makes one thing clear: matching benchmark averages does not equal matching performance in real agentic work. The intelligence that compression erases is precisely what gets exposed in scenarios requiring long-range planning, self-monitoring, and sustained forward progress.

Practical Takeaways

This test leaves local model users with a few memorable lessons:

• Bonsai largely matches full-precision models on short tasks, but is nearly unable to complete long-chain builds — compression works in the former case and fails in the latter.
• If you're running agents locally, always read the trace logs. They'll tell you directly where the agent is getting stuck and help you improve the agent loop.
• On MacBook, use Prism ML's llama.cpp fork, because it caches conversations between steps; as of the time of recording, the MLX version does not cache, making it noticeably slower.

The conclusion is straightforward: for chat, quick code writing, and short tool calls on a laptop, Bonsai 2 is a genuinely impressive model — compression really does work in these scenarios. But for long-running, unsupervised agentic work, you should still reach for a full-precision model.

Compression technology lets us run large models on local machines, but it doesn't fully preserve a model's real intelligence. When you're planning to benchmark a quantized local model against a larger full-precision version, that's something worth keeping firmly in mind.