AI Agent Ran All Day for Just 1 PR? A Postmortem of Three Time-Swallowing Mistakes

Title

AI Agent Ran All Day for Just 1 PR? A Postmortem of Three Time-Swallowing Mistakes

Meta Description

A developer reviews an AI Agent workday: only 3 of 14 attempts accepted, attributed to three mistakes—not logging attempts, environment drift, and lacking scoring criteria—with reusable fixes: an attempt ledger, preflight environment checks, and build/scope/proof gates.

Article Content

Writing code with an AI Agent and getting only one merged diff out of a whole day—this is the reality for many developers. In the form of "a letter to myself from yesterday morning," one developer reviewed a day of Agent runs that involved 14 attempts with only 3 accepted, and attributed it to three mistakes ranked by time lost. Based on the original record, this article breaks down these three traps and the directly reusable remediation scripts.

Mistake One: Recording Only Results, Not Attempts

The original author's self-diagnosis is blunt: he remembered the few successful runs and forgot the eleven failed ones. That's not data—it's emotion.

Without an attempt ledger, you can't distinguish "the model is too weak" from "the harness is broken," nor can you compare two models without guessing. The author's fix is a pure standard-library Python script, attempt_ledger.py, that appends a line of JSON per attempt, with fields including attempt number, model name, verdict (accept/reject/error), wall-clock time, token count, and failing gate.

The script's key constraint: write immediately after each attempt, and don't batch-record at the end of the day. Another rule matters just as much—a reject must also carry a gate name, one of build, scope, or proof; no gate, no record row.

The author also stresses that the tok/acc in the summary output (tokens per accepted result) should be read as a cost column. The sample output in the original shows only the format shape, not measured results—keep that in mind when citing it.

Attempt ledger draws on changelogs in software engineering and lab notebooks in experimental science; the core idea is that every Agent run is an experiment with a hypothesis and must leave traceable observations, not just the accepted results. It's typically implemented as an append-only JSON Lines file, one record per line, naturally suited to script-based statistics and after-the-fact diffs. The difference from CI logs is that the ledger records the basis for decisions (which gate failed, how many tokens were consumed), not the full execution output, so it's small and can be kept long-term. Without a ledger, developers can only attribute failures from memory, easily mistaking environment issues for model capability issues.

Mistake Two: Treating the Runner Machine as a "Snowflake"

The sixth attempt died on a lockfile mismatch. The problem wasn't the prompt—the machine had drifted relative to yesterday.

This is the most tempting part of the free-server option, and also where the hidden risk lies. The author's approach is to freeze the environment before the Agent runs: first generate a baseline on a clean tree with sha256sum package-lock.json > env.lock, then write a preflight.sh that verifies the environment lock and checks whether the Git working tree is clean before each attempt.

This preflight blocks three kinds of failure: stale lockfiles, uncommitted changes, and re-running on old state. The author estimates each one cost him over an hour.

Order is the whole point—preflight runs before the prompt, not after the diff.

"Snowflake server" is a classic anti-pattern in infrastructure: every machine is unique like a snowflake, configured by hand-accumulated, irreproducible setups, and hard to troubleshoot once it drifts. The opposite is the "phoenix server" or immutable infrastructure—machines can be rebuilt at any time from code and lockfiles. Although the Runner here is just a free server, the same principle applies: the sha256 of package-lock.json is the environment fingerprint, and preflight moves the "immutable" constraint forward to before each run. Understanding this makes clear why a lockfile mismatch isn't a minor issue but a signal that the environment has slipped out of version control.

Mistake Three: Reviewing Diffs Without Scoring Criteria

"Looks right" isn't a gate—it's a coin flip with extra typing. The author says every diff he accepted on gut feel later turned into a review comment—the most expensive way to learn.

His solution is three gates and a verdict, checked in order, rejecting at the first failure:

Gate	Question	Failure action
build	Does the test suite pass?	Record with --gate build
scope	Does the diff touch only allowed paths?	Record with --gate scope
proof	Is there a new test that fails before the change and passes after?	Record with --gate proof

The most frequently skipped of the three is proof. The concrete approach: run the new test against the old code first, watch it fail, then accept the diff. The author believes this step turns "the model seems good" into a claim that can stand up in review.

The method behind the proof gate is test-first, derived from TDD's red-green-refactor cycle: write a test that fails on old code (red), then accept the change that makes it pass (green). In the Agent context, the point is to validate causality—the new test's failure proves it actually covers the behavior to be fixed, rather than being a tautological empty test. A common pitfall is checking only that "the new test passes," which can also hold on old code and thus loses discriminating power. In practice, you can require the Agent to include the failing test's output in the prompt as evidence for the proof gate.

Free Tier Is Not a Universal Answer

The original explicitly frames the free tier as a budget choice, not a general solution, and provides a boundary table: low-blast-radius scenarios like temporary refactors and comparing two candidate approaches suit free models plus free servers; scenarios involving code that can't be sent to third parties, release gates that require determinism, and latency-sensitive interactive tools do not—because data handling is in your own hands, free capacity is best-effort rather than SLA, and queueing behavior propagates to user experience, respectively.

The author's advice: write down the boundaries before you start. Making this decision at the sixth hour is exactly how a day disappears.

The Complete Loop: A Seven-Step Cycle

1. Commit a clean code tree
2. Run preflight.sh
3. Include the failing test as prompt input
4. Apply the diff
5. Run the three gates
6. Record the verdict and failing gate
7. Repeat; run summary only at the end of the day

The author estimates the whole setup takes about ten minutes, with the rest of the time spent on attempts. By his account, mailing this letter to his 9 a.m. self would cut a day's cost from eight hours to four.

What This Method Can't Solve

The original ends with restraint: a ledger won't make a weak model strong, it only lets you see the truth sooner; free capacity is best-effort, so don't put an immovable deadline behind it; no scoring criteria can replace actually reading the diff—it replaces remembering the diff.

One addition: the MonkeyCode free quota cited in this article (free model access for the writing step, free server option for the running step, 10 million token quota on the free tier) comes from the original author's own account, and that article is MonkeyCode product promotion. Free-tier quotas, server specs, and terms change, so before building a week's workflow around it, confirm the current terms on the project page.

One-line Summary

A developer reviews 14 Agent attempts with only 3 accepted, attributing three mistakes ranked by time lost and providing remediation scripts.

Paragraph Summary

In the form of "a letter to myself from yesterday morning," the article dissects a day of 14 AI Agent coding attempts with only 3 accepted, ranking by time lost three mistakes: recording only successful results rather than all attempts, treating the Runner machine as a drifting "snowflake," and reviewing diffs on gut feel without scoring criteria. For each mistake, the original offers directly reusable fixes: an append-only JSON attempt ledger via a pure standard-library Python script, freezing the environment before the prompt with a sha256 lockfile plus a preflight script, and running build/scope/proof gates in order, rejecting at the first failure. It also gives a seven-step loop and makes clear that free models and free servers suit only low-blast-radius scenarios—code that can't be sent externally, deterministic release gates, and latency-sensitive tools are all out.