Five Pitfalls and Fixes for a Self-Auditing Local Agent Loop

When an Agent Grades Its Own Homework: Real Lessons from a Local Loop

A locally deployed drafter/critic agent loop "completed" a task for the first time, and nothing on disk had changed. The drafter reported success, the critic signed off, and the task was marked done — but the working directory was identical to when it started. The issue took a full week to track down, and the cause turned out to be a piece of dead code the author had written and which had never once executed.

The design of this loop was very clear: a drafter model with real file access makes changes in the task directory, and a separate critic model judges against acceptance criteria. If the critic rules it a failure, the drafter retries with the errors the critic pointed out. After consecutive failures hit the iteration cap (three by default), the task is escalated to blocked for a human to handle. Two independent models judging each other, plus a hard cap to prevent infinite loops — on paper this is entirely reasonable, but running it on a single 24GB GPU exposed a large number of problems.

Pitfall One: The Critic You Think You Have Isn't a Real Critic

By design, the critic was supposed to be "a direct structured-output call with temperature set to 0" — pure judgment, no tool calls, with the return value enforced by a schema. That's the whole point of the role: a judge that issues a verdict, not an agent that goes off and does things.

But the code didn't do that. The critic's model label was read from the critic's agent record, and that record always existed. So the flag deciding whether to "dispatch as an agent" or "dispatch as a structured call" was always true. The structured-output branch that was supposed to enforce the verdict schema became dead code that never ran.

The consequence: the critic ran inside exactly the same CLI harness as the drafter, with tool permissions. A small model handed a tool harness doesn't behave like a judge — it behaves like an agent. Instead of returning a verdict, it tried to read the prompt file — emitting a read tool call, claiming "I can use the Read tool," and then failing to parse. The loop retried, failed again, and eventually gave up.

The fix was to add a setting that made the structured path reachable again. But the real lesson is: the role you describe in your docs is not the role your code enforces. The critic was conceptually read-only, but nothing at the dispatch layer guaranteed that.

Structured output means constraining a model's return value to predefined fields and types via a JSON Schema or similar, commonly used where reliable judgments are needed. By contrast, tool calling lets the model, beyond generating text, initiate actions like read and write, which the harness parses and executes. These are fundamentally two different dispatch modes: the former treats the model as a side-effect-free function, the latter as an agent that can act on its environment. When the dispatch flag is hardcoded to true, the model always takes the tool-calling path, even when the caller expects pure judgment. In practice, such flags often live in agent record metadata, coupled to the role label, so a single misuse leaves every downstream branch permanently unreachable, silently degrading into the wrong mode.

Pitfall Two: The Judge Wrote the Very Files It Was Supposed to Judge

This is a case anyone building autonomous pipelines should know about. In one run, a complete Angular project scaffold suddenly appeared in the working directory — 23 files, including everything from angular.json to package.json. The task explicitly forbade the use of frameworks and package managers.

The drafter didn't do it. The action diff in the audit log (computed from real before-and-after directory snapshots, which a model can't forge) attributed those 23 files to the critic model. In the same dispatch, it both returned a clean {"verdict":"pass"} and wrote an entire application into the directory it was supposed to judge impartially.

The cause was exactly pitfall one: when the critic was dispatched through the CLI agent, it got exactly the same execution environment as the drafter — the working directory, full write permissions, the flag to skip permission confirmation. A model that decided to "build the thing first, then check it" just went ahead and did so.

The fix that ultimately made this structurally impossible: routing the critic to a pure text-judgment call with no subprocess and no working directory — a component with no write capability at all. A read-only judge that can't touch the artifacts isn't a nice-to-have; it's the dividing line between a trustworthy verdict and a self-contradiction.

The action diff in the audit log is typically derived by comparing before-and-after snapshots of the working directory, rather than relying on the model's self-report. Such diffs are trustworthy because they compare the actual state of the filesystem: which files were added, modified, or deleted can all be objectively recorded, and the model can't forge or conceal it. Common companion metrics include the duration of each model call, token counts, and exit codes. When diagnosing attribution questions like "who touched the file," snapshot-based diffs are often more reliable than model outputs or natural-language logs, because they bypass the model's self-description layer and directly observe side effects. This is also the key evidence chain that let 23 Angular files be attributed precisely to the critic model in this case.

Pitfall Three: Model Choice Quietly Determines Whether Files Get Written

After fixing the runtime, the drafter ran twice, reported success, and wrote nothing. The audit log showed why: a model the author had labeled "coder" emitted fabricated <tool> blocks as plain text rather than making real tool calls. The harness saw text, executed zero actions, and nothing changed on disk. The model described the process of writing files, but never actually did it.

This model had been assigned based on file size and reputation. The audit log actually already contained the evidence needed to predict this failure — statistics on which models produced real file operations through the harness. Two models produced hundreds of real operations, and the other two produced zero. The author hadn't consulted the logs before reassigning.

The systemic flaw is that the agent dropdown offers a model simply because the local server pulled it, with no probing of tool-use capability. A drafter using a model that doesn't support tool calling will produce confident but hollow attempts across the entire iteration cap, reporting success the whole way.

The harness is the runtime shell surrounding the model, responsible for parsing model output into real actions: recognizing <tool> blocks, invoking the corresponding functions, and feeding the results back as context for the next round. Its behavior depends heavily on whether the model natively supports the tool-calling protocol. If the model merely "writes out" a tool block in text, the harness treats the whole thing as an ordinary string, executes nothing, and so the loop appears to make progress and reports show success while the disk stays untouched. This class of failure is the hardest to diagnose, because the model's self-account is entirely plausible, and only the "zero real file operations" statistic in the audit log reveals the truth. So before wiring any model into an agent loop, it's worth running a probe task to verify it can actually drive the harness.

Pitfall Four: Loading Overhead Ate 81% of the Time

The drafter/critic loop alternates roles constantly within a single run — drafter, critic, drafter, critic. On a local runtime, every model switch can mean a cold load. In the first run, 81% of wall-clock time went to model loading, not actual generation. The runtime was configured to keep only one model resident and pinned permanently. The fix was just two environment variables, but the insight is that the bottleneck was the runtime configuration, not the model, and not the GPU.

Model residency and cold loading are key configuration items for local inference. When using two models, a drafter and a critic, if the runtime can only keep one model in VRAM, every role switch requires unloading the old model and reloading the new one from disk or memory. 24GB of VRAM is usually not enough to hold two medium-sized models simultaneously, so the switching cost gets amplified over and over until it dominates wall-clock time. Solutions typically include: keeping the runtime with only one model resident and pinned, or adjusting residency policy, quantization precision, or concurrency limits via environment variables. Notably, this kind of bottleneck shows up in monitoring as a large share of loading time, not GPU compute saturation, so you have to look at runtime configuration before suspecting the hardware.

What Actually Worked

It wasn't all a tale of things going wrong. The auto-generated acceptance criteria were genuinely strong — 19 items, each traceable to the task description, locked in without prompting. The state machine never produced an illegal transition, even after repeated dispatch failures. Observability was what made all of this debuggable in the first place: a JSONL audit log for every model call, containing duration, token counts, and most crucially the real directory snapshot file-action diff. Nearly every number in this article comes straight from that log.

When the author re-ran the pipeline after the fixes, it delivered a seven-file site with zero human intervention — and when checked from outside the system with a script that never consults the loop, 64 of 66 checkpoints passed.

The Most Counterintuitive Lesson

Declaring is not verifying. The drafter claimed it wrote files. The critic claimed it judged the files. The task claimed it was done. None of those claims was backed by what actually happened on disk — until the author reconstructed the entire execution from the recorded run state and directory snapshots. The strongest signal in the whole system, status: done, turned out to be nothing more than a copy of a single step's verdict, not a check on the overall task.

If you're building a local agent loop, there are three things worth doing before you trust it:

1. Enforce the judge's role at the dispatch layer, not just in the docs. A read-only critic that can't write is the difference between a trustworthy verdict and a self-contradiction.
2. Probe the models for the capabilities you actually use. Tool calling and structured output are two different capabilities. "It's a coding model" is not evidence that it can drive your harness.
3. Treat runtime configuration as the number-one suspect. When 80% of your time goes to loading, the problem is almost never the model itself.