Stop Auditing the Agent, Audit the Counters: An AI Coding Workflow Retrospective

Introduction: A Failed AI Coding Experiment

At 08:20 I opened two terminals and didn't close them until 19:40. One was running a free server, the other an Agent writing patches. By lunchtime I had accumulated three diffs and couldn't tell which one came from which run.

This wasn't a model problem—it was my lack of counters. This article retrospects those five wasted hours and presents a reproducible six-step workflow that has been verified to survive a full week.

What It Actually Cost That Day

I spent about three hours re-running the loop because I couldn't match diffs to run records. Another two hours went to debugging a patch that a reviewer model had "blessed." The rest was consumed by a fixture directory whose hash changed on every run.

All three failure types point to the same root cause: missing counters. Agent retries, wrapper retries, shell loop retries—three layers of retry sharing a single log file with no timestamps. That's not observability; it's noise.

Mistake One: Letting the Harness Count for Me

The most fatal habit was letting the harness record attempt counts. The Agent, the wrapper, and the shell loop each retried independently, and the log held only a single result line with no timestamp.

The fix is plain: write the ledger row before the first model call, not after the last. The ledger is a TSV file—boring but effective.

#!/usr/bin/env bash

**What is a ledger (run ledger)?**

In data engineering and databases, a ledger originally referred to an append-only record of transactions. Its core property is that once a record is written it is never modified; new events can only be appended to the end. The difference from an ordinary log is that a ledger emphasizes "serving as evidence"—it records what happened, not what you happened to print while debugging.

In AI Agent workflows, a ledger is typically a structured table file (such as TSV/CSV) where each row represents a traceable event: run ID, stage name, sequence number, input hash, elapsed seconds, and result status. TSV is used instead of JSON logs because it's boring enough: no nesting, no schema migration issues, and `tail`, `awk`, and `grep` can process it directly without any extra log aggregation service.

The author's emphasis on "writing the ledger row before the first model call" reflects a key design principle of ledgers: record intent before recording results. If you only write a row after success, then crashes, timeouts, and interrupted runs leave no trace at all—yet those are precisely the failed runs that most need to be tracked. Writing "attempt 1 start" first, then the result, is what lets you reconstruct after the fact "did this run actually happen, and how far did it get?"
# Adapted example. Not run as-is; substitute your own paths and commands.
set -euo pipefail

RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)"
ART="./runs/$RUN_ID"
mkdir -p "$ART"
LEDGER="./ledger.tsv"

log() { printf '%s\t%s\t%s\t%s\n' "$RUN_ID" "$1" "$2" "$3" >> "$LEDGER"; }

FIXTURE_HASH="$(find fixtures -type f -print0 | sort -z \
  | xargs -0 sha256sum | sha256sum | cut -d' ' -f1)"
log fixture "$FIXTURE_HASH" "frozen-input"
log attempt "1" "start"

My rule now is: no ledger row, no model call. An unrecorded run, even if the diff looks correct, is just hearsay.

Mistake Two: Treating the Reviewer's Approval as a Test

I once had a model check a patch, it said "looks correct," and I merged it. The unit test then failed on an off-by-one error in a range check.

The reviewer model produces opinions; the test suite produces exit codes. Only the latter can block a merge in CI.

The fix: the reviewer only sees the diff and the failure output—nothing else. Feed it the minimum information needed to explain the defect.

**Why is the exit code a harder signal than natural language?**

The exit code is the integer a Unix process returns to its parent on termination, by convention 0 for success and non-zero for failure. CI systems (such as GitHub Actions and GitLab CI) decide whether a pipeline passes or fails based on the exit code, not on text in the logs. This means a model can say "looks correct" in natural language, but it cannot forge a test suite's exit code—unless the test itself actually passes.

The asymmetry here is worth noting: a reviewer model's output is a probabilistic natural-language judgment, potentially influenced by prompt phrasing, residual context, or even diff formatting; whereas the output of a test framework like pytest is deterministic program behavior. When the two conflict, trusting the deterministic signal and treating the uncertain signal as a secondary clue is the engineering default.

Restricting the reviewer's input to "diff + failure output" is also a way to reduce noise: giving a model more context doesn't always make it judge more accurately—it may instead follow unrelated code and fabricate a rationale. The minimum-information principle is often more reliable than "give it the whole repo" in review-type tasks.
# Adapted example: reviewer input is diff + failing test output only.
git diff --no-color > "$ART/patch.diff"
pytest -q 2>&1 | tail -n 40 > "$ART/test_tail.txt" || true

# Build the reviewer payload from those two files, not from the repo.
printf '%s\n' 'Review this diff against this test failure.' \
  'Return JSON: {"verdict":"agree|disagree","reason":"..."}' \
  '--- DIFF ---' > "$ART/review_prompt.txt"
cat "$ART/patch.diff" >> "$ART/review_prompt.txt"
printf '%s\n' '--- FAILURE ---' >> "$ART/review_prompt.txt"
cat "$ART/test_tail.txt" >> "$ART/review_prompt.txt"

Then compare two things: the reviewer's verdict and the test exit code. When they disagree, stop immediately and have a human read the run record.

Mistake Three: Writer and Reviewer Sharing the Same Workspace

The writer Agent wrote temporary files, and the reviewer read a polluted directory. The next run's fixture hash didn't match the ledger, and the diff lost all provability.

The fix is three hard constraints: directory isolation, a hash gate, and a write-path allowlist. The gate must run before any shell command executes, not after.

**What is a fixture hash gate?**

A fixture refers to the fixed input data or environment snapshot that a test run depends on—for example, sample files, database seed data, or a pre-provisioned directory. If the fixture is modified during a run, then the same command produces incomparable results at different times, and the diff loses its meaning—you can't tell whether the difference comes from a code change or from input drift.

The hash gate works like this: at the start of a run, compute a deterministic hash over the entire fixture directory (sort the files, sha256sum each one, then hash the aggregate result), and write that hash into the ledger; before each shell command execution, recompute it and compare against the ledger value, aborting on mismatch. The word "deterministic" is crucial here: `find ... | sort` guarantees stable file ordering; otherwise the hash changes every time and the gate produces false alarms.

The gate must run before command execution, not after: checking afterward only detects pollution—it cannot prevent a polluted run from producing a seemingly legitimate diff. Combined with directory isolation (separate workspaces for writer and reviewer) and a write-path allowlist, the three together form the precondition for "this run can be proven."
# 1) Plan must be small, typed, and declare its write paths.
jq -e '
  (.steps | length) > 0 and
  (.steps | length) <= 5 and
  has("write_paths") and
  all(.steps[]; (.cmd | type) == "string")
' plan.json > /dev/null

# 2) Write paths must stay inside the allowed trees.
jq -r '.write_paths[]' plan.json | while read -r p; do
  case "$p" in
    ./src/*|./tests/*) ;;
    *) echo "refusing path: $p" >&2; exit 3 ;;
  esac
done

# 3) Fixture pack must be unchanged since the ledger row was written.
NOW="$(find fixtures -type f -print0 | sort -z \
  | xargs -0 sha256sum | sha256sum | cut -d' ' -f1)"
[ "$NOW" = "$FIXTURE_HASH" ] || { echo "fixture drift" >&2; exit 4; }

The Six-Step Loop That Survived a Week

1. Freeze the fixture pack and record its hash first, before anything else.
2. Write the plan as JSON and validate it with jq and the allowlist gate.
3. Run the test suite on the free server, capturing the exit code and tail output.
4. Send only the diff and failure tail to a free model for a second read.
5. Compare the reviewer's verdict against the exit code; abort on disagreement.
6. Append one ledger row per step, including actual elapsed seconds.

The six steps produce evidence you can replay on Friday. That's the whole point.

Where Free Capacity Doesn't Apply

Free access is an availability statement, not a throughput promise. I plan around short bursts, not deadlines. I also make no assumptions about persistence—artifacts get pulled back locally immediately.

Scenario	Local Only	Free Server	Paid Fixed SLA
Single-file fix, tests under 30s	Suitable	Optional	Overkill
Needs system packages or services	Slow	Suitable	Suitable
Unattended multi-hour loops	Risky	Risky	Suitable
Involves credentials	Never	Never	Isolated runner only
Second read of diff	Suitable	Suitable	Optional

Three rows in the table are judgment calls; two rows are hard constraints rather than preferences.

Limitations and Who Shouldn't Use This

Free model access and free server options are both statements from their operators; verify the terms yourself. I haven't measured throughput, and I won't schedule release plans on it.

If your tests require production credentials or regulated data, skip this workflow. If you need contractual uptime or data residency guarantees, skip it. If human reviewers can't read the ledger before a merge, skip it too.

Advice to 08:20-Me

Write the counters before the prompts, write the hash before the diff. When the reviewer and the tests disagree, trust the tests and call a human.

If you want to try it, run the six-step loop with free model access and the free server option. Even if you drop both tomorrow, keep the ledger.

Disclosure: This article was written as part of a promotion for the MonkeyCode product.