Same Plugin, 86.1% or 8.5% Token Savings? Measuring Accurately Comes First

"How many tokens does this plugin save?" is a question that is very easy to get wrong. The same plugin, on the same 400-case workload, saves 86.1% of billed tokens at one threshold, and only 8.5% at another. Both numbers are true, but neither means anything without its conditions.

dsh-phocinae is a local decision plugin for the DeepSeek^Ad Harness (dsh) Agent. It provides two tools, phocinae_ask and phocinae_gate, backed by a small model that can answer structured decision questions in a single forward pass. The workload measured here is the public typed-decisions row set: 400 cases, 2000 decisions, with Phocinae-Largha-150M-v1 (CPU, fp32) as the local model and flash-tier deepseek-flash (temperature 0) as the hosted model of comparison. This article breaks down how these numbers were actually measured.

Why "Token Savings" Is Hard to Measure

The hosted model side has the API's own usage counters; local decisions themselves are not billed, but that does not mean zero cost — their tool definitions are sent with every request, and each call's arguments and rendered results enter the transcript, where they are billed as usual on the next request. So-called "savings" always equal "not sent," and you must first specify what exactly you are counting.

What makes it thornier is that the escalation unit is not the "decision." Once an escalation fires, the hosted model receives a whole case — one state plus its five structured questions. If even one decision is uncertain, the entire case gets re-answered from scratch. So "the number of decisions kept local" and "the number of cases that never touch the API" are two different statistical lenses, and they yield two different percentages.

The threshold itself is a product decision. What the plugin outputs is an escalate signal, not a savings guarantee. τ (below which confidence is marked as uncertain) is chosen by the caller, and it is precisely where the 86.1% row and the 8.5% row diverge.

Confidence Threshold (τ) and the Escalation Signal

What the plugin outputs is not a binary "answer / don't answer" pair but a calibrated confidence score. τ is the dividing line: decisions scoring below τ are marked escalate and handed to the hosted model for re-answering; those above τ stay local. This mechanism turns "should I spend tokens?" into a knob the caller can adjust. The subtlety is that τ is not an internal parameter of the plugin but a product-level trade-off — raising τ means "rather spend more than answer wrong," lowering τ means "rather take some quality risk to save." The difference between 86.1% and 8.5% in the article is essentially the same model giving two answers under two risk preferences, not two contradictory measurements. Only by understanding this can you make sense of why every number below carries a τ footnote.

The Pre-execute Hook vs. Message Billing Distinction

The article notes that the approval gate "runs as a pre-execute hook, not a message, so its increment to session tokens is zero." Behind this is an easily overlooked billing boundary: hosted APIs typically bill only content that enters the message sequence (roles like system, user, assistant, tool), whereas a hook is logic the client triggers locally before a tool actually executes — it produces no output tokens and does not enter the next round's context. So phocinae_gate's 212-token tool definition counts toward per-request fixed overhead, but its decision process each time it intercepts a command has zero impact on the bill. Conflating the two will overstate the approval gate's cost by a good margin.

Measurement Design: Three Counters

Per request — the fixed part. While the plugin is loaded, the two tool definitions appear in every request's tool list: phocinae_ask at 473 tokens plus phocinae_gate at 212 tokens, i.e. 685 tokens per request. This is counted with the vendor's own tokenizer, not estimated. The approval gate is the exception: it runs as a pre-execute hook, not a message, so its increment to session tokens is zero.

Per decision — the round-trip overhead. One local call's arguments are 99 tokens and the rendered result is 135 tokens, giving 234 tokens per decision in steady state; if this is the first decision in the session, the fixed 685 is added on top.

Per case — the granularity at which the hosted model is actually invoked. Across 400 cases the hosted-side usage totals 1,222,211 tokens (290,606 input, 931,605 output — roughly 76% of the bill goes to the model writing its own answers), i.e. 3,055.5 tokens per case. The local side, by construction, is zero-billed — a decision is a forward pass, not a generation.

Why Count with the Vendor's Own Tokenizer

The article repeatedly stresses that the figures 685, 234, and 3,055.5 are "counted with the vendor's own tokenizer, not estimated." This matters especially in cross-model comparisons: different model families have very different tokenization rules, and the same English passage can differ by 10% to 30% in token count across GPT-family, Claude-family, and DeepSeek-family models. If you use one estimator to count plugin overhead locally while reading the API's returned usage on the hosted side, the two lenses won't line up, and the "savings rate" will gain or lose a chunk out of thin air. Unifying both counters on the hosted model's tokenizer is the only way to guarantee that "tokens saved" and "tokens on the bill" are measured with the same ruler.

Where Per-Decision vs. Per-Case Billing Diverges

The article distinguishes "the number of decisions kept local" from "the number of cases that never touch the API," and the root cause is that the hosted model's invocation granularity is the case, not the decision. A case contains one state plus five structured questions; if any one of those decisions is escalated, the entire case must be re-sent to the hosted model for a fresh answer — meaning that even if 95% of decisions stay local, if they are scattered across 95% of cases, virtually every case will be escalated. So decision-level savings (say 92%) and case-level savings (say 86%) are not a simple conversion; it depends on whether escalations cluster or spread out. The reason τ=0.6 performs so poorly is precisely that the confidence distribution is concentrated near the median of 0.6294, so escalated decisions are sprinkled across a large number of cases, triggering the "one drags down the whole case" cascading effect.

Results: Every Number Comes with Conditions

The pure-local path. All decisions resolved locally, zero escalations. Compared with handing everything to the hosted model, 1 decision saves 69.9%, 10 decisions save 90.1%, and 1000 decisions save 92.3%. It saturates around 92% rather than 100%, and the gap is exactly that fixed counter: tool definitions and round-trip overhead always remain in the caller's context.

The τ=0.4 hybrid mode is the row this article is more willing to recommend on this workload. Of 400 cases, 27 are escalated (6.8%), and billed tokens drop from 1,222,211 to 82,499. Counting the plugin's own overhead (685 per request + 234 per decision kept local), the savings rate is 86.1%. Accuracy is 0.7305, versus 0.7085 for hosted-only and 0.7320 for local-only — on this workload, escalation is nearly a wash.

The τ=0.6 hybrid mode is the "honest row," and also the plugin's default factory value. Of 400 cases, 363 are escalated (90.8%); looking only at the model side saves 9.2%, and counting plugin overhead brings it to just 8.5%. Accuracy also drops to 0.7115, below local-only's 0.7320 — because escalated cases are re-answered whole, and the hosted model is weaker on this workload's selection-style questions (0.6793 vs. the local model's 0.7529).

Why does 0.6 escalate so much? The confidence distribution is concentrated in the middle, with the median decision landing at 0.6294, and a single uncertain decision is enough to drag down an entire case. Of 2000 decisions, 851 (42.6%) fall below 0.6, which by case count is 90.8%. A threshold that sounds conservative is, on this distribution, practically a full opt-out.

At the paired public unit prices ($0.28 per million input, $1.10 per million output), 400 cases hosted-only costs $1.1061, τ=0.4 costs $0.0747, and pure-local incurs no billed tokens.

What Was Not Measured

The 400 machine-readable cases are a bench, not a conversational trace. Real sessions have longer histories, more tools, and more turns; the next step should be to aggregate API usage with the plugin on and off on the same 30+ task list, while also tallying escalations, false blocks, and task success rates. Until then, the numbers above can only be treated as a calibrated bench, not a promise.

All latency data comes from CPU, fp32, single-machine: local decision HTTP round-trip 42.6 ms, approval gate command screening averaging about 21 ms, whitelisted commands 0.02 ms. The circulating GPU figures (fp16, p50 21.0 ms on an RTX 5090) are the model publisher's numbers, not this measurement, and have not yet been reproduced. τ=0.4 is not a universal constant either — it is simply a product of this workload's confidence distribution.

Reproducing on Your Own Traffic

All scripts and raw results are kept under the repository's bench/. One command reruns everything: first start the local model with python -m phocinae.main, then point node bench/token-savings.mjs at your own rows and model pair, then use context-cost.mjs to count the plugin's own overhead, economics.mjs to compute installment and hybrid rows, and gate-by-mode.mjs to test the approval gate in isolation. Sweep τ across {0.3, 0.4, 0.5, 0.6} and read accuracy and savings side by side: on this workload, pushing τ higher saves less and actually hurts accuracy.

Whether a savings claim is credible comes down to whether you can rerun it. What is given here is: τ=0.4, 400 cases, including plugin overhead 86.1% — and, because "an unreported knob is an ununderstood knob," the 8.5% at τ=0.6 alongside it. Point the bench at your own logs, then trust your own row.