Same Model, 0 to Perfect Score: Agent Reliability Hinges on When You Call the Model

When an agent underperforms, many developers' first instinct is to swap in a bigger model. But a recent experiment on a real Android device shows that the same 4B small model jumped from 0/20 to 20/20 success simply by changing when the model gets consulted. This raises a key question: how much of an agent's reliability comes from the model itself, and how much comes from the structure around it?

Experiment Design: Injecting Failures on Real Hardware

The developer turned an Android phone into a screen-on-always device and injected a variety of failures: app crashes, hung background services, unavailable backends, and even simulated human interference like "someone picking up the phone." Each failure triggers a recovery loop: observe, decide, policy gate, execute, verify. The decider can be swapped between pure rules, rules-first with model fallback, or fully model-driven. Each scenario runs 20 times on real hardware, with an independent verifier checking the results.

To test the value of structure itself, the developer deliberately chose a locally running 4B small model rather than a frontier LLM. The core question: given limited model capability, how much improvement can good structure deliver?

4B small model refers to a language model with roughly 4 billion parameters, typically runnable locally on a phone or laptop with extremely low inference cost, but noticeably weaker in generalization and reasoning than frontier models with 10B+ parameters. Local execution means no cloud API dependency — low latency, good privacy, offline-capable, but limited model capability. Agent here means a program that perceives its environment, makes decisions, and executes actions to accomplish recovery tasks, typically driven by an "observe—decide—execute—verify" loop. Policy gate is a filtering layer in the decision flow: before executing an action, it checks rules or conditions to decide whether to let it through, blocking high-risk or non-compliant operations. In the experiment, swappable decider means that under the same perception and execution framework, you can switch between pure rules, rules-first with model fallback, or pure model as the decision backend — isolating the two variables of "structure" and "model capability."

Rule-Induced Failures: Knocked Out Before the Model Even Steps In

The initial failure case was classic: when someone opened another app, a rule — "another app is in the foreground, pull the display UI back" — fired before the model was consulted, causing the agent to wrongly interfere with the user, yielding 0/20 success. The problem wasn't the model; it was the rule stepping in too early.

The developer introduced a contention mechanism: if evidence the rule can't read (e.g., someone touched the screen in the last 30 seconds) contradicts the rule's action, the rule is considered contested, and the model is consulted first. With the same model, success jumped straight to 20/20. Meanwhile, the control scenario (app in foreground but no touch) still achieved 20/20 with zero model calls. This shows it wasn't that the model got smarter — it's that the model was summoned at the right moment.

Contention mechanism is a meta-rule: when the evidence a rule relies on is incomplete or weakened by contrary evidence, don't execute the rule directly — hand the decision to the model instead. In implementation, each rule declares the evidence types it depends on (e.g., screen touch events, foreground app state), and the system checks before triggering whether that evidence is available and uncontradicted. If a touch is detected within the last 30 seconds but the rule only sees "another app in foreground" and can't read the touch evidence, the rule is flagged as "contested" and routed to the model inquiry flow. This design shifts the model's invocation timing from "fixed fallback" to "triggered by evidence conflict," reducing unnecessary model calls while preventing rules from misjudging when information is insufficient.

Rules First: The Model Speaks Only When Rules Are Silent

The experiment also found that a rules-first strategy matched the pure-model approach's accuracy in almost every scenario. For predictable failures, the pure-rules approach achieved a perfect 240/240 recovery. The model only proposes, while a deterministic policy layer adjudicates: only actions in the catalog are allowed, the least destructive option is preferred, and budgets and cooldowns are enforced. No irreversible action is authorized on the model's "confidence" alone.

This design positions the model as an "advisor" rather than a "commander" — leveraging its generalization while backstopping with deterministic policy to avoid the risks of model hallucination.

Deterministic policy layer is an adjudication module with no randomness or model inference, filtering and ranking candidate actions by fixed priorities and constraints. Typical constraints include: action catalog (a whitelist allowing only predefined actions), least destructive first (e.g., try restarting a service before considering clearing data), and budget and cooldown (limiting action frequency per unit time to prevent flapping or storms). Irreversible actions mean operations that can't be rolled back — deleting data, uninstalling apps, factory reset, etc. The policy layer requires such actions to have extra authorization or human confirmation and won't release them based on model output confidence alone. This way the model only provides candidate suggestions, and final execution is decided by auditable deterministic logic.

Verification and Learning: Evidence Beats Confidence

The experiment stresses verifying results with evidence the decider can't see. On a real phone, a force-stop command returns success while the app is still running — "command returned OK" doesn't mean "the problem is fixed." Only independent verification can confirm whether the failure is truly resolved.

The learning mechanism is equally restrained: recurring answers get compiled into rules, but candidate rules must pass replay over 484 historical events before being promoted. What really keeps bad rules out of the system is replay verification, not the model itself. This forms a "propose—verify—solidify" closed loop that lets the rule base evolve steadily without spiraling out of control.

Independent verifier is a check module separate from the decider, using observation signals the decider can't see (e.g., whether the app process is still alive, whether the UI is interactive, backend health check results) to determine whether the failure is truly fixed. It's isolated from the decider to avoid the bias of "verifying itself." Replay verification means re-executing candidate rules against the historical event log and checking whether their decisions match known-correct outcomes; only full or high-proportion passes are allowed to promote a rule to official status. Historical event replay is essentially offline regression testing, catching rules that work in a single scenario but misfire in others. The 484 historical events in the experiment form the replay dataset; rules must pass here to be promoted, forming the "propose—verify—solidify" closed loop.

Takeaway: Check Your Structure Before Reaching for a Bigger Model

The core message of this experiment: before reaching for a bigger model, check when your agent gets consulted and what checks its output. A stronger model is just the next run away — the model itself is merely a configuration switch. For teams building reliable agents, investing in rule boundary design, contention handling, and independent verification processes may yield more direct gains than upgrading the model.

The developer's closing question is worth pondering for every practitioner: in your system, how do you decide the boundary between rules and the agent?

One-line Summary

With the same 4B small model, just changing when it's consulted raised success from 0/20 to 20/20 — structure matters more than the model.

Paragraph Summary

The article covers a developer's experiment injecting app crashes, service hangs, and human interference failures on a real Android device to test agent recovery with the same 4B small model. Initially a rule misfired before the model was consulted, yielding 0/20; after introducing a "contention mechanism" that consults the model first when a rule's evidence is contradicted by signals like touch, success became 20/20. The experiment also shows a rules-first strategy achieving 240/240 recovery on predictable failures, with the model only proposing and a deterministic policy layer adjudicating, an independent verifier checking real fixes, and candidate rules needing to pass replay over 484 historical events to be promoted. The core conclusion: agent reliability comes more from structural design — rule boundaries, contention handling, and independent verification — than from model size, so check your structure before swapping in a bigger model.