Hands-On with Ternary Bonsai 2 9B: 3x Faster, But Not Ready for Production

A Replacement That Looked Like a Bargain

The production classifier runs on a single RTX 2080 Ti with 11 GB of VRAM. During peak hours, requests frequently overflow to a hosted model, and the cost adds up. When PrismML released Ternary Bonsai 2 9B, the team saw a practical opportunity: the GGUF file is only 2.25 GiB, versus 6.70 GiB for the 27B they were running—nearly a threefold difference.

But size and speed advantages alone aren't enough to justify a direct swap. The team replayed 1,015 production decisions. On laptop benchmarks, the 9B was 2.4–3.6x faster than the 27B—yet it downgraded 15 of 65 escalation requests that had already been confirmed by a secondary model. On another task, 43% of answers violated application-layer rules. That result alone killed the idea of a simple replacement.

The Real Pressure of Production

The existing 27B runs on llama.cpp with PQ2_0 quantized weights, two parallel slots, and a shared 32k context. Reasoning is disabled, responses go through strict json_schema decoding, and temperature is set to 0. Clients have an 8-second deadline; if the gateway can't allocate a slot within that budget, it rejects the request and falls back to the hosted model.

Over a 20-hour window, the system logged 9,081 requests: 82% were served by the local model, 13% were fast-rejected for missing the deadline, and 4.6% were canceled by clients. During the busiest hour, roughly 30% overflowed to the hosted model. The median model time for served requests was 4.3 seconds, with a p95 of 8.1 seconds.

The application needs to make three types of judgments on informal, non-English posts. Each request includes the current post and up to 11 historical posts, with median prompt lengths between 0.7k and 2.3k tokens, and responses capped at 64–96 tokens of JSON. Task A is escalation triage, outputting allow / clarify / escalate plus a severity subtype and confidence score; missing an escalate is the most costly error. Task B is intervention assessment, deciding whether intervention is needed and suggesting an action. Task C is a multi-flag detector that outputs six booleans.

Constrained decoding (JSON schema decoding) refers to applying a grammar or structural mask to the model's output logits at each step of autoregressive generation, zeroing out the probabilities of tokens that don't match the target format, so that the final result is guaranteed to be accepted by the parser. Runtimes like llama.cpp implement this via JSON schema or GBNF grammars. Its key limitation is that constraints only guarantee "format validity," not "semantic coherence." When a schema allows a set of fields to be freely combined, while the mutual exclusivity the application actually needs is only specified in the natural language prompt, a small model can easily generate JSON that parses but is absurd in business terms. This is exactly why "intervention needed: true + continue as normal" appears in Task B—structurally valid, logically contradictory, and only catchable through an additional application-layer validation.

How the Replay Test Was Done

The team sampled from production logs of the current prompt version, deliberately preserving rare outcomes. For Task A, they replayed all 107 escalations (65 confirmed, 42 rejected), 200 clarify cases, and 250 allow cases. For Task B, they replayed 175 each of "intervention needed" and "continue." For Task C, they replayed all 108 logged cases.

To estimate production rates, each case was weighted by sampling and logging probabilities: weight = (rows in stratum / rows sampled) × (1 / logging probability). The Task A sample represented roughly 68k production checks. Requests were reconstructed using the application's own prompt builder, with stored posts and thread windows, the same schema, token limits, and temperature. Across 275 inspected cases, prompt token counts matched production exactly.

Replays ran on an M1 Max, with the 27B using production build prism-b10709 and the 9B requiring prism-b10777. Since logged answers for the 27B already existed, only 275 cases were rerun: a random control subset plus all Task A and C disagreement cases. The 27B reproduced 273 of 275 production decisions; the two edge differences likely stem from Metal/CUDA numerical differences.

Stratified sampling with inverse probability weighting is a common statistical technique for evaluating rare events. In production logs, high-cost but low-frequency outcomes like escalate may only be a few percent; if sampled at natural ratios, there would be too few positives in the replay sample, causing enormous estimation variance. The approach is to first stratify by outcome (stratum), sample at different rates within each stratum, and then assign each sample a weight weight = (total rows in stratum / sampled rows in stratum) × (1 / logging probability), so that the weighted distribution again approximates the true production distribution. The logging probability term corrects for the bias introduced by "not all requests being fully logged." Be aware that weighting restores expected proportions but cannot compensate for insufficient sample size—in this article, Task C has only 5 none cases, and even with weighting, the false positive rate cannot be reliably estimated. This is precisely why "agreement looks okay but evidence is too thin."

Task A: More Ambiguity, Less Clarity

Task A's performance is the most telling. The 9B always produced valid JSON—the problem was the decisions inside.

Production 27B Decision	n	9B Agree	9B Differ
escalate (confirmed by hosted model)	65	50 (76.9%)	15 → clarify
escalate (rejected by hosted model)	42	34 (81.0%)	8 → clarify
clarify	200	184 (92.0%)	12 → allow, 4 → escalate
allow	250	141 (56.4%)	109 → clarify

The worst part is those 15 downgraded confirmed escalations. The 27B reproduced all of them as escalate in local reruns, 12 with the most severe explicit subtype and 3 with the metaphorical language subtype. The 9B assigned low-severity subtypes to all 15, with self-reported confidence between 0.85–0.9. In other words, confidence thresholds simply can't catch these errors.

The team read through these cases themselves; about 11–12 looked clearly positive, and 3–4 were borderline. Some stated key signals directly; others used metaphor or idiom. This was an informal review, not expert annotation.

The bigger shift was a concentration toward clarify. After reweighting by production volume, allow dropped from 86.4% for the 27B to 49.5% for the 9B, while clarify rose from 13.4% to 50.1%. Weighted agreement was 61.2%. The 9B never turned an escalation directly into allow, but it pushed a large number of two-sided cases toward the middle. Since clarification has a cost in the application, this affects a substantial portion of users. It also doubles the weighted escalation rate; the new escalations from the clarify stratum alone would add roughly 180 hosted-model reviews over the log period.

Task B: Contradictions Inside Valid JSON

Task B exposed the limits of constrained decoding. Of 350 answers, 151 (43%) returned "intervention needed: true" alongside "continue as normal." The application rejects this combination outright.

Production 27B Decision	n	9B Agree	9B Invalid	9B Opposite
Intervention needed	175	23 (13.1%)	128	24
Continue	175	148 (84.6%)	23	4

The JSON schema allows this combination; the prompt asks the model not to use it. The 27B followed this instruction in control runs; the 9B often didn't. Constrained decoding guarantees parseable JSON but cannot enforce a relationship not written into the schema. Even looking only at valid answers among positive cases, agreement is just 23 out of 47. After reweighting, about 17% of Task B checks would fail validation and fall back to the hosted model—actually increasing the load the team was trying to reduce.

The impact of quantization on model capability is typically not uniform. Extremely low-bitwidth schemes like PQ2_0 and ternary introduce errors in weights and activations that tend to have less effect on surface capabilities like "format adherence," because constrained decoding and language priors are enough to hold up valid output—but they have a more pronounced effect on capabilities requiring deep reasoning, such as "logical consistency across multiple fields." You can think of it this way: the combination of reduced model size and quantization error makes the 9B more likely to generate each field independently on Task B rather than maintaining constraints between fields. The team was also aware of this confound—they didn't run a full-precision Qwen3.5 9B, so they can't distinguish whether it's "the 9B is too small" or "ternary quantization is too aggressive." Separating these two effects would require a controlled replay across multiple quantization levels of the same model.

Task C: Decent Agreement, But Too Little Evidence

Task C performed best of the three. The 9B agreed on 91 of the 103 cases the 27B flagged as flagged or urgent (88.3%). Disagreements included five flagged cases downgraded to none and three urgent cases downgraded to flagged.

But the team had only five none cases—too few to estimate false positives. This task raises fewer concerns, but the evidence is correspondingly much thinner.

How Much Faster?

Running llama-bench with flash attention on an M1 Max:

	pp512, tok/s	tg128, tok/s
27B, prism-b10709	134.1	19.2
27B, prism-b10777	142.9	19.1
9B, prism-b10777	484.5	45.6

Compared to a fixed 27B build, the 9B is 3.6x faster at prefill and 2.4x faster at generation. Updating only the runtime had almost no effect. On the production 2080 Ti, the 27B measured 749 tok/s prefill and 42.6 tok/s generation. Scaling by the laptop speed ratio, inspection time could drop from 4.3 seconds to about 1.5–2 seconds. But the team didn't actually benchmark the 9B on the 2080 Ti, so this remains an estimate.

What to Try Next

This test answered whether the 9B could replace the 27B with existing prompts, not what the best small model could do. The prompts were tuned for the 27B, reasoning was disabled for both models, and only one domain in one language was tested. They also didn't run a full-precision Qwen3.5 9B, so they can't separate model size effects from ternary quantization effects.

The first thing worth changing is Task B's output schema—merging the boolean and the action into a single enum so the contradictory combination becomes structurally impossible—then tuning the prompt for the 9B and replaying the same cases. A small reasoning budget might also help, at the cost of some speed.

Another direction is using the 9B as a first stage for Task A, passing only non-allow answers to the 27B. On this sample, about half of production checks could skip the larger model, missing about 6% of 27B clarify cases but no escalations. The team hasn't tested this as a pipeline; fitting both models on an 11 GB card would be tight, and the second hop would add latency.

For now, the 27B stays. Replaying production traffic made this decision straightforward: the speedup is real, and so are the missed escalations. Any replacement must pass the same test—no missed confirmed positives, no increase in invalid answers.

Cascade inference (initial screening + review) is a common compromise architecture when replacing a large model with a small one: use the cheap model for most traffic and send only "uncertain" or "specific-category" requests to the large model. The design question is who gets through the first hop. The team here chose a conservative strategy—all non-allow outputs from the 9B go to the 27B—meaning they'd rather run an extra hop than let the first hop miss an escalate. The cost is that hosting both models in 11 GB of VRAM is tight, and every second-hop hit adds inference latency. Another approach is the reverse: let the 9B only release "high-confidence allow" cases and send everything else to the 27B. The choice depends on the business's relative tolerance for false negatives and latency.

One-line Summary

Ternary Bonsai 2 9B is small and fast, but it misses confirmed escalations and generates self-contradictory answers—so the team kept the 27B.

Paragraph Summary

To reduce hosted-model costs, the team tested replacing their production 27B with Ternary Bonsai 2 9B: about 3x smaller, 3.6x faster prefill and 2.4x faster generation on a laptop. But replaying 1,015 production decisions revealed: on Task A, the 9B downgraded 15 of 65 confirmed escalations to clarify while assigning them high-confidence (0.85–0.9) low-severity labels, so confidence thresholds couldn't catch them; meanwhile many cases shifted toward clarify, with weighted agreement only 61.2%. On Task B, 43% of answers contained the self-contradictory combination "intervention needed" + "continue as normal," which constrained decoding couldn't block. Task C performed better but had too few samples. The team ultimately decided to keep the 27B and proposed next steps like merging output enums and a 9B-first, 27B-review cascade.