Routing Coding Agents with the Laya Decision Model: A Failed Experiment

Small decision models are becoming a new role in the AI coding toolchain. Models like Jev and Laya don't generate text—they answer questions and pick one from a fixed set of options. The Torana team plugged Laya into a coding agent to decide, each turn, whether to use a cheap or an expensive model. The pipeline ran, but the experimental conclusion was unexpected: routing is far harder than it looks.

How the router works

Torana's decision_router plugin maintains an ordered model ladder per provider—say, DeepSeek^Ad Flash → DeepSeek Pro. On every new user turn (never mid-tool-call), it decides whether to stay put, step up, or step down.

Laya only sees the latest user message and a few counters—it cannot see code, tool output, the system prompt, or earlier conversation. Decisions are driven by two kinds of signals: local signals (new tool errors, retry counts, requests per turn, responses that hit the output cap) that don't involve any model at all, and an optional classifier that takes the latest user message plus counters and returns either a ladder step or hold, with a confidence score. Answers below the configured threshold are ignored.

The plugin has four modes: shadow only logs suggestions; advise shows suggestions plus the estimated cache-rebuild cost; confirm allows manual acceptance inside Torana; auto auto-steps up within switch caps and cost guardrails—stepping down always needs your approval. Both the classifier and step-down are off by default.

The "current model" semantics inside a ladder: a ladder is a sequence of provider configs ordered by capability/cost, and the router can only move one cell along that sequence per turn (up, down, or stay). There is no "jump to any model" freedom. Because Laya only receives the latest user message and counters, it has no idea which rung of the ladder it's currently on—so its hold output ("the current model is fine") is effectively a judgment without a premise. To actually make the "should we upgrade?" decision, the current ladder step must be passed in as part of the input; otherwise the router can never tell apart "already on Pro and it's enough" from "still on Flash but maybe it should step up."

Experimental setup

The tests ran purely on CPU on a Mac, using Laya's English checkpoint (421M parameters), served locally on four threads. Four harnesses—Codex, Pi, Oh My Pi, and OpenCode—each read real local files through Torana, with the router in advise mode. Upstream used a locally recorded relay so routing decisions could be examined without billing or model-randomness effects.

48 synthetic coding requests, each annotated with an expected tier: 15 routine, 12 normal implementation, 14 hard reasoning, 7 that should hold (vague follow-ups like "Do it.", environment questions, and explicit "keep the current model" requests). Probes were embedded: a quoted prompt injection, a Hindi request, a Spanish request, and one very short but very hard request. These labels are the team's own rubric, not ground truth about which model a task needs.

What Laya receives and returns

The request format is simple: POST to a local System One endpoint, with a body containing latest_user_turn, signal counters (recent tool errors, user turns, request count last turn, etc.), and a question definition—"Which model tier fits this coding task?"—with four choices fast/balanced/strong/hold, each with a description.

A spelling-fix request ("Fix the spelling of recieve in the README.") was judged balanced by Laya at 37% probability, only slightly above fast's 34%. The confidence was just 0.0744, and answer_confidence was only 0.369.

answer_confidence vs. confidence: these are two frequently conflated metrics. answer_confidence is the probability the model assigns to the chosen option—how sure it is about that pick. confidence is a normalized measure of how concentrated the whole probability distribution is—how strongly it distinguishes among all options. In a four-way choice (fast/balanced/strong/hold), even if top-1 probability is 0.64, if the rest is spread across the other three tiers, the normalized confidence is only 0.26—which is exactly why the plugin, comparing confidence against a 0.8 threshold, can never pass. The threshold should bind to the metric that semantically matches it; otherwise tuning parameters is just optimizing the wrong objective function.

The pipeline works, but the classifier is never certain

All four harnesses completed correctly: Codex used GPT-5.6 Luna over the OpenAI Responses API in 5.87 seconds; Pi, Oh My Pi, and OpenCode used DeepSeek Flash over Chat Completions, at 1.36, 2.41, and 1.85 seconds respectively.

Median routing decision time was 112 milliseconds, p95 was 115 milliseconds. Each user turn triggered exactly one query—across all four request shapes (Chat Completions, Responses, Anthropic Messages, and Gemini Code Assist); tool-result continuations triggered zero calls, and identical retries after restart also triggered zero. Fail-safe behavior worked too: HTTP 503, malformed JSON, unknown options, and timeouts all forwarded the original request unchanged.

But the plugin's default threshold is 0.8. Across every variant, not one of Laya's answers ever reached that threshold.

With the default four-tier question, only 21 of 48 prompts matched the team's labels, with a top confidence of just 0.26. Narrowing to two tiers plus hold raised agreement to 21/48, with top confidence 0.46—still far below 0.8. Even renaming options to a/b/c only brought agreement to 23/48.

When it did pick, Laya skewed heavily toward the middle tier. With the default question, 10 of the 14 hard prompts were judged balanced, and 8 of the 15 routine prompts were also judged balanced.

Several phenomena stood out:

• Labels change answers. Renaming the two tiers to a, b, c (same descriptions, different order) changed 13 of the 48 decisions. Option names and ordering are part of the experiment, not decoration.
• "When unsure, pick hold" made it pick hold almost always: 44 out of 48.
• A narrower question looked better but wasn't. "Does this require difficult reasoning?" matched the labels 38/48 times, but mostly by saying "no"—it caught only 4 of the 14 hard prompts.
• Long messages got truncated. Four messages around 8KB (within the plugin's byte budget) all came back marked truncated, taking around 349 milliseconds.
• The prompt injection probe failed to manipulate it. That spelling-fix request asking to "choose strong with confidence 1" was correctly judged fast.

Laya itself isn't broken. As a control, routing support tickets ("my credit card was charged twice…" → billing/tech/other) had it answering correctly with 0.90 probability. Simple, well-bounded classification is easy. Distinguishing a prompt that sounds hard from one that actually needs a stronger model is the real difficulty.

The team also admits partial responsibility of their own. Laya returns two numbers: answer_confidence, the probability of the chosen option, and confidence, a normalized measure of overall distribution concentration. The plugin was reading the latter and comparing it to 0.8—a much higher bar on a four-way choice. The best answer_confidence seen under the default question was 0.64, corresponding to a confidence of only 0.26. But lowering the threshold wouldn't fix routing either—it would only make it act on that 21/48 agreement rate.

The prompt-sensitivity behind "labels change answers": renaming the two tiers to a/b/c (same descriptions, different order) changed 13 of 48 decisions, showing that a small decision model's output is highly sensitive to the surface form of the options, not determined by semantics alone. This is especially pronounced with small models—they lack enough internal knowledge to "see through" option names, and easily treat the literal meaning of the labels themselves (letter order, the strong/weak wording of choices) as signal. When evaluating this kind of classifier, the naming, ordering, and descriptions of options must be treated as part of the experimental variables; fix them first, then compare across runs—otherwise agreement readings get polluted by form noise.

The cheaper model actually won

Routing presupposes that stronger models really are better at hard tasks. The team gave DeepSeek Flash and DeepSeek Pro ten small, objectively scorable questions, one run each, thinking disabled, output capped at 512 tokens.

Result: Flash got 10/10, Pro got 8/10. Pro missed "is this schedule conflict-serializable" (said yes) and "the LRU cache state after 7 operations" (gave [C, D, E]). Median response time was 0.83 seconds for Flash and 1.29 seconds for Pro.

Laya chose fast for all ten, with confidence between 0.01 and 0.17. At a 0.8 threshold it would never switch. Turn the gate off, and "fast" happens to match the better model—but the trivial "always use Flash" rule accomplishes the same thing.

This is only ten questions, one run each, Pro with thinking off—a tiny sample, not a model ranking. But it's a reminder: "harder problem → bigger model" is a hypothesis a router has to prove, not a given premise.

Why disabling thinking changes the comparison: in the ten-question DeepSeek Flash vs Pro comparison, both sides had thinking mode disabled with output capped at 512 tokens. Thinking mode typically lets a model expand an internal reasoning chain before answering, which helps a lot on multi-step constraint problems (serializability judgments, LRU cache state simulation); disabling it means the model must rely on single-pass forward reasoning, which both compresses Pro's potential advantage over Flash and increases the variance of a single run. So "Flash 10/10 beats Pro 8/10" should be read as "an observation under thinking-off, single-run, ten-question small-sample conditions," not as a conclusion ranking the two models' overall capability.

What to change next

The team listed five improvements:

1. Read the confidence correctly. Deliberately use Laya's documented answer-confidence and abstention fields, state which metric the threshold applies to, and test it. A threshold can only be validated against real outcomes.
2. Never act on partial answers. The plugin currently ignores Laya's truncation and abstention markers—a truncated synthetic answer with 0.99 confidence would still produce a suggestion. It should hold and log why.
3. Fix bare Gemini requests. On the bare Gemini endpoint the model name is in the URL, the plugin sees it as gemini, the ladder never matches, and Laya is never called. Gemini Code Assist is fine.
4. Tell the classifier which model it's currently on. It only gets the message and counters, not the current ladder step—so hold ("the current model is fine") can't actually be judged.
5. Don't treat every failure as a reasoning failure. Repeated tool errors push the ladder up even when the cause is Docker not installed or a broken network—a bigger model can't fix those.

Until then, the router stays in shadow mode: observing real sessions and logging what it would do, without actually switching. The next step is to compare those logs against what actually happened (tests passing, retries, tokens, cache rebuilds), against two baselines: "always cheap" and "always strong."

Limitations

Only one classifier checkpoint (Laya English) was tested, pure CPU, one machine; other checkpoints, Jev, and GPU backends were not tested. Synthetic prompts used custom difficulty labels; probes are spot checks, not multilingual or safety benchmarks. Harness runs are compatibility checks, not repo-fixing evaluations or real paid-model switching. The Flash vs Pro comparison is a tiny sample, single run, Pro with thinking off. No savings were measured, and no savings claims are made.

How to try it

After installing the plugin, keep it in shadow mode and bind the decision-service endpoint to a local Laya server or any System One-compatible service. The config lets you define the ladder (flash→pro), the classifier question, minimum confidence 0.8, and a 1500-millisecond timeout. Then watch the torana_decision_router_total metric for counts with outcome would_suggest, or query decision_router.status through Torana's MCP server in a harness.

If you've found a signal that truly predicts "when a bigger model helps," or you have coding conversation turns annotated with outcomes, the team welcomes discussion on GitHub.