Don't Let AI Grade Its Own Work: Why Self-Reviewed Code Always Passes

Letting the Agent that writes the code decide whether it's done almost always results in a pass—this isn't a matter of attitude, it's structurally determined. The original author hits on the most easily overlooked flaw in current AI programming workflows with a clear point: when the person writing the code and the person accepting the code are the same, acceptance is bound to be distorted.

Why Self-Evaluation Is Doomed to Fail

The Agent that wrote this code retains "what I intended to do" in its mind. When it checks back, it compares code against intent, not code against results. So "does this code run?" quietly becomes "is this what I envisioned?"—and the answer is of course always yes, because the code was generated according to its vision.

A more insidious layer is the incentive direction. For the same model, saying "done" means the task is closed, while saying "not yet" means work continues to accumulate. Without role isolation, the shortest path always wins. The original calls this "by construction"—it's not that the model wants to be lazy, but that the process design itself rewards self-passing.

Writing and Verifying Must Be Split Into Two Roles

The original lists four dividing lines between "author" and "reviewer," which constitute operable role isolation principles.

Reviewers Don't Read Code, Only Run Commands

The reviewer role executes tests, runs lint, runs builds, calls endpoints. Its subjective opinion of the code doesn't count—only the exit code counts. This line pulls acceptance from "does it read like it's right" back to "does it run right," eliminating the space for aesthetic judgment and intent comparison.

Scores Are Numerical, With Hard Thresholds

Scoring range is 0-100, with a pre-declared threshold (e.g., 85). "I think it's fine" doesn't pass; below the threshold is failure. Numericalization allows acceptance conclusions to be recorded, compared, and traced, rather than remaining at vague qualitative descriptions.

The exit code repeatedly mentioned in the text is the integer status returned to the caller when a Unix process ends—0 typically indicates success, non-zero indicates various failures. Test frameworks, linters, and build tools all encode results into exit codes, making them naturally suitable as machine-decidable acceptance signals: no need to parse human-readable output text, no need for the model to "understand" logs. This also explains why reviewers only run commands rather than read code—exit codes compress complex quality judgments into a deterministic boolean value, eliminating hallucinations and leniency that language models might produce when interpreting logs.

Failures Must Include Reasons

Saying only "failed" isn't enough—the report must list specific failure items and corresponding command output. Without failure reasons, the fixing role can only blindly modify in the dark, easily rewriting parts that weren't broken.

Reviewers Don't Fix

Review only measures, doesn't fix. Fixing is done by the author, then returns to the same ruler for re-measurement until it passes. This constraint avoids the slippery slope of "changing standards while testing."

What It Looks Like in Actual Operation

The original provides a set of comparison outputs:

SCORE=100.0  THRESHOLD=85.0  GATE=passed
SCORE=60.0   THRESHOLD=85.0  GATE=failed
GAPS=["pytest: exit code 1, expected 0"]

The same loop, the same code, two rounds yield two different answers—the reason is that a deliberately broken test was placed in the second round. The reviewer doesn't know this—it just measures. This controlled experiment shows that when acceptance authority is given to a role that only recognizes exit codes, scoring has discriminative power.

What Changes in the Workflow

After roles are split, the focus of daily collaboration shifts from "listening to the Agent report" to "looking at report data." The original summarizes three changes.

No longer reviewing promises. The deliverable is a report with scores and reasons, not a statement of "successfully implemented." Reviewers face verifiable evidence, not claims whose credibility they must judge themselves.

Agent self-improvement has a target. With clear failure reasons in hand, it only needs to fix the broken parts rather than tearing down and rewriting. Fix scope narrows, and side effects narrow with it.

Regressions become visible. A score drop between rounds means something broke—visible before going live. Scores become a continuous health curve rather than a one-time pass/fail.

Applicable Boundaries and Remaining Issues

The premise of this method is that "correctness can be verified by commands." Projects with high test coverage benefit directly; but for dimensions that can't be expressed by exit codes—UI aesthetics, product logic trade-offs—numerical review still falls back to subjective judgment. The original doesn't elaborate on this part, which is something practice needs to fill in.

Another implicit cost is setting the review threshold: too low, and review is merely nominal; too high, and the Agent gets stuck in repeated fix loops. The 85 given in the original is just an example value; real projects need to adjust based on their own test maturity.

The original author also provides a skill link called AI Test Loop and a paid guided configuration service (running together for the first cycle), indicating this pattern has been packaged into a reusable workflow. For teams, it's worth running a round on your most mature test suite first to see whether scores can really distinguish "runs" from "runs correctly."

Background Supplement

The "threshold" here corresponds engineering-wise to the concept of quality gate. A quality gate refers to presetting a set of quantifiable criteria in the process, which must all be met to proceed to the next stage—commonly seen in CI/CD pipelines as code coverage thresholds, zero high-severity vulnerabilities in static scanning, etc. Applying it to Agent self-evaluation, the key isn't the number itself but "pre-declaration"—once the threshold is set, it cannot be renegotiated during review, otherwise when reviewer and fixer share the same goal, standards drift with results. In practice, thresholds should be set based on historical data, e.g., taking the lower edge of scores from recently manually-accepted samples, then leaving some margin, rather than picking an integer by feel.

The principle behind "the same ruler" is measurement immutability: the commands, configuration, and test cases used in review must remain frozen between fix rounds—only the code under test changes. If tests are modified or assertion strength lowered during fixing, a score increase no longer represents quality improvement but merely that the measurement tool was relaxed. This aligns with the control group concept in scientific experiments—only with a single variable can conclusions be attributed. Engineering-wise, this can be enforced through hash verification of review scripts, mounting test suites as read-only, etc., ensuring every round's score falls in the same coordinate system.

One-line Summary

Writing code and accepting code must be split into two roles, otherwise Agent self-evaluation almost always passes.

Paragraph Summary

The article points out that letting the code-writing Agent judge whether a task is complete almost always results in a pass—this isn't an attitude problem but determined by process structure: the same Agent compares code against its own intent rather than code against runtime results, and "saying done" is the shortest path incentive-wise. The solution is to split writing and verifying into author and reviewer roles, with four dividing lines: reviewers don't read code, only run commands, with exit codes as the criterion; scoring uses 0-100 numerical values plus preset hard thresholds; failures must include specific command output; reviewers only measure, don't fix. Controlled experiments show that under the same loop, introducing a deliberately broken test drops the score from 100 to 60, proving that scoring has discriminative power only after role isolation. The article also notes this method applies to correctness verifiable by commands, while dimensions like UI aesthetics still require human judgment, and threshold setting needs adjustment based on project test maturity.