144.3M Local Decision Model vs Hosted API: An Approval Gate Selection Guide

A significant portion of the transactions that coding agents handle in the background aren't really "thinking" — they're permission judgments: Can this command run? Should this ticket go to finance or customer support? Is this output safe to publish? Each is a small, structured decision. If you use a chat API to handle them one by one, the cost shows up in three places — token consumption per call, network round-trip latency, and decision context leaving the local machine.

This article uses Phocinae-Largha-150M-v1 (144.3M parameters, Apache-2.0 license) as an example to lay out a selection logic for approval gates: what should stay local, what goes to a hosted model, and the numbers and caveats on each side.

What a Decision Model Actually Does

A decision model doesn't chat. You give it a state (a short description of the situation) plus typed questions — noul (yes/no), choice (pick one from N), score (2–10) — and a single forward pass returns answers and confidence for each question. The answer set exists before the call: nothing is generated, so nothing drifts.

In the model card's own usage example, one decision reports input_tokens: 146, output_tokens: 0 — zero output tokens, because there's no output text. This is precisely its key value in the "gate" scenario: deterministic, auditable, replayable decisions.

State and typed question are the two basic inputs of this protocol. A state is a short text description of the situation, like "user requests deleting files under /var/log" or "this ticket mentions a refund and an invoice number"; typed questions constrain the model's output space — noul only allows yes/no, choice picks one from N pre-given options, score falls in an integer range of 2–10. Crucially, each forward pass receives one state plus up to five questions at once, returning two fields per question: answer and confidence. Confidence isn't the model's self-reported probability; it's a calibrated column (see the ECE note below) that can be used directly for thresholding. The fundamental difference from a traditional chat API is that the output distribution is pre-constrained by the question type — the model has no room for free interpretation, so there's no wording drift, formatting errors, or hallucinated text. This is exactly the formal basis for auditability and replayability.

Three Questions to Help You Choose the Right Track

1. How often does this decision trigger?

If the same type of decision appears dozens of times a day, calling a hosted API is a good trade — low frequency, diverse, convenient. If it triggers thousands of times a day — approvals, triage, screening, routing — the arithmetic changes shape. Local forward passes have no per-call billing; the cost shifts to hardware you control and doesn't grow with call volume.

2. Can the input leave this machine?

An approval gate might see command lines, file paths, log tails, sometimes customer content. If policy or common sense requires that context to stay local, the decision must be made locally. This is the hardest of the three filters, because there's no room for negotiation.

3. Is the answer set closed?

If the possible answers can be enumerated in advance — allow/ask/deny, queue A or B, a severity score from 2–10 — a typed decision model fits. If the answer is truly open — "write an incident summary," "explain this stack trace" — keep the large model. The dividing line isn't model size; it's whether the answer already exists before you call.

One practical footnote on latency and hardware: the published p50 under GPU fp16 is 21.0 ms per decision (RTX 5090). On CPU it's slower — 1.64 s single-threaded per case (one state plus five questions), about 8–20 decisions/sec with 8 threads. This is the domain of background gates, not interactive typing latency.

The Numbers, Including the Awkward Ones

Readings under the same protocol, each with its own annotation — all from the public model card:

Reading (typed decisions)	Value	Release notes
Largha-150M-v1, en	0.906	400 cases / 2000 decisions; expert model fitted on that dataset's training set; model card notes this is above the 0.735 teacher self-consistency reference; the dataset card flags scores far above the reference as at risk of label-specific overfitting
Laya	0.766	Self-measured, native interface
JEV	0.727	General model, zero-shot
meraGPT	0.768	General model, zero-shot

Lineage note: the fine-tuning mix includes a 500-item error-correction pool mined from the model's predictions on the typed-decision test rows. The 0.906 we report is a fitted reading, not a zero-shot result.

There's also the awkward number that needs saying alongside it: JevBench public-231 reading is 0.5455 (126/231) — below its 58.4% acceptance threshold, a fail. This number comes from a public 231-item set; the measurement environment and protocol differ from the 400-case self-evaluation above — the two cannot be merged. The model card publishes it honestly, and those JevBench rows were excluded from training.

There's also a Chinese reading: 0.848 — measured on machine-translated cases, while the training mix itself contains roughly 2,400 machine-translated Chinese rows and roughly 1,400 native Chinese rows. Treat it as a fitted Chinese result, not zero-shot transfer.

ECE (Expected Calibration Error) measures whether, when the model says it's 0.8 confident, the actual accuracy is really near 0.8. Lower values mean confidence is more trustworthy; higher values mean the model's self-reported confidence is out of sync with actual performance. In this article, the shipped confidence column has an ECE of 0.2519, meaning the model often isn't sure when it says it's "very sure" — if you directly use that column for threshold gates, you'll let through many uncertain decisions by mistake. The built-in recalibration column brings ECE down to 0.0168, relying on temperature scaling: introducing a scalar temperature constant before softmax to re-stretch the logits distribution so probabilities better match the true hit rate. The temperature constant is fitted on the validation set, and the project ships the constant and the columns needed for calibration. So in practice, any τ-based escalation gate must first go through recalibration, otherwise the threshold is meaningless.

Hybrid, Not Replacement: Escalation Gate Design

The approach that keeps both sides honest is a confidence gate. Each decision returns a calibrated confidence; at τ=0.6, the gate escalates 45.0% of official-set decisions, reducing LLM calls by 55.0%, and the retained subset accuracy rises from 0.906 to 0.9936 (full-route accuracy 0.8135). But the actual small print on the model card says: "τ was scanned on the evaluation set — it needs to be re-scanned per domain."

So local vs hosted isn't either/or. The small model handles steady-state traffic; the hosted model only receives calls the gate flags as uncertain, novel, or high-risk. Escalation isn't failure — it's routing.

An honest detail you need to know if you're building on this gate: on the model card, the shipped confidence column has an ECE of 0.2519 (en); the built-in recalibration column brings it down to 0.0168. If you're setting thresholds based on confidence, recalibrate first — the project ships that column and the temperature constant needed to complete calibration.

The principle of the confidence gate and τ threshold is simple: each local decision returns a confidence; if below τ, escalate to the hosted model, otherwise accept directly. τ=0.6 is the operating point scanned on the evaluation set, and its effects — 45.0% escalation, 55.0% reduction in LLM calls, retained subset accuracy 0.9936 — show that escalation isn't partial failure, but a routing mechanism. Two things to watch: first, τ must be re-scanned per domain; confidence distributions differ across healthcare, finance, and operations, and directly copying τ=0.6 will be imbalanced. Second, the escalation rate itself is an operational metric; a 45% escalation rate means nearly half the traffic still goes to hosted calls, which directly determines the cost structure. Full-route accuracy 0.8135 refers to overall performance when no gate is set and all decisions are adopted in a mix — it's lower than the retained subset's 0.9936 because the high-confidence decisions the gate lets through still contain errors, reminding us that improving retained subset quality doesn't equal improving whole-system quality.

Must-Read Caveats Before Selection

• Tool routing is a weak point. The project's server confuses semantically close tool names — a reproducible case in the model card returns fetch_url instead of web_search at about 0.33 confidence. Low-confidence choices should be escalated, never taken as final results.
• Not a safety judge. The model card's exact words matter in the gate scenario: use as a first-line signal with escalation — paired with your own rules and human paths, never as the sole guard.
• Long-document reasoning is out of scope. Base context is 8192 tokens; the model card's "what it is not" list says "not for long-document reasoning." Keep the gate's inputs to states and questions, not articles.
• These are measurements under a single setting. Latency and quality numbers come from the release run; your hardware and traffic will differ, so it's worth measuring yourself.
• The model card publicly self-corrects. Training overlap disclosure was corrected on October 10; current numbers were re-measured on current weights, with a revision history. Read the disclosures before citing any single number — including mine.

Tool routing is a weak point directly related to model size. With 144M parameters, distinguishing among a set of semantically similar tool names — like web_search vs fetch_url, refund_order vs cancel_order — requires precisely slicing highly overlapping semantic neighborhoods in embedding space, and small models tend to latch onto one shared keyword and make a judgment on fine-grained comparisons like this. The reproducible case on the model card returned the wrong tool at about 0.33 confidence. Fortunately, low confidence is a visible signal: as long as you strictly follow the rule "low-confidence choices must be escalated and must not be executed directly," tool-routing errors can be caught by the gate. This is also the value of the escalation path repeatedly emphasized in this article — the weak point doesn't need to be eliminated, only identified and routed away. Any deployment that directly adopts local tool-routing results without a confidence check will amplify this type of error.

What's Worth Trying, and What Isn't

Scenarios suitable for a local gate:

• Decisions are structured and repetitive — command approval, ticket triage, output screening, document routing;
• Input cannot leave the machine — local deployment, air-gapped environments, regulated contexts;
• The per-call billing shape is wrong, or you want steady-state offline availability;
• Or you want a first-line gate that escalates uncertain cases, rather than answering everything.

Scenarios that keep the hosted model:

• The answer is open-ended text — summaries, explanations, long-document reasoning;
• Decisions are rare and diverse, with no pattern to amortize;
• Or the call itself is the escalation tail — the part the gate explicitly refuses to guess.

pip install phocinae-server
hf download Phocinae/Phocinae-Largha-150M-v1 --local-dir ./largha
PHOC_MODEL_DIR=./largha python -m phocinae.main   # 127.0.0.1:8155

A summary from third-party benchmarking is worth keeping: 42 real decisions, Jev (hosted) overall 71.4%, while Clef-Flash (9B, running locally on his 3090) got 66.7%. His framing is worth remembering: this exercise doesn't tell you which model is best, it tells you which calls are wasting money.

Transparency statement: this article was drafted with AI assistance; all numbers come from publicly linked sources, and no new benchmarks were run for this article.