AI Agent Self-Audit Ledger: 10% of Records Come from a Test That Never Fails

An open-source AI Agent is supposed to "verify itself" — every artifact it produces gets executed and checked, and each gate run's result is appended as a line of JSON to a local ledger. But when we audited our own ledger, we found an unsettling truth: 10.1% of records come from a test that has never once failed.

The Ledger Mechanism: No Self-Scoring

This open-source Agent, called OpenAmer, uses a strict verification mechanism. After each artifact is produced, it's actually executed, and the raw output of the check — including exit codes and error text — is written directly into the ledger's why field. There's no self-scoring, no fuzzy "pass/fail" judgment, only verifiable execution results.

The ledger format is simple: one JSON object per line, containing a timestamp, task, goal, Agent identifier, pass/fail status, and a why field holding the verbatim output of the check. The repo provides a public snapshot at docs/EVIDENCE-LEDGER-SNAPSHOT.json, while the live file on the laptop lives at memory/si/outcome_ledger.jsonl.

The design philosophy deserves credit: a ledger you carefully curate isn't a ledger, it's marketing material. But the problem emerges precisely from this honestly-recorded data itself.

The semantics of a gate: In the context of continuous integration and Agent self-verification, a "gate" refers to an automatable admission check — only when the check passes is the artifact allowed to proceed to the next stage or be marked as valid. The strength of a gate determines how much information the ledger carries: a genuine gate should be able to distinguish "correct behavior" from "incorrect behavior," so its output space contains at least two results, pass and fail. If a gate has never returned a failure across a large number of historical runs, there are usually two explanations — either the subject under test really is stably correct, or the gate's assertion is so loose that it can't catch any defect. OpenAmer's ledger records the gate's raw output (exit code and error text) verbatim into the why field, precisely so readers can judge after the fact which of these two this gate actually is, rather than just seeing a flattened pass-rate number.

The Deceptiveness of Aggregate Numbers

As of the latest measurement, the ledger contains 1,852 lines of gate-run records, of which 579 passed and 1,273 failed, for an aggregate pass rate of 31.3%. The time span runs from September 28 to October 13, 2026, covering 223 distinct goals, 833 tasks, and 79 Agents.

31.3% is not a vanity number. But averages conceal the whole truth. When the 1,852 lines are grouped by goal, the problem surfaces.

One Goal Occupies 10% of the Ledger and Never Fails

After grouping by goal, the most striking finding is this: the goal gX has 187 lines of records, 100% passing, with only 1 distinct task. These 187 lines account for 10.1% of the entire ledger, all recorded by the same Agent on the same task, with the same why field every time:

exit=0 OK

Across 187 runs from October 4, 2026 to today, it has never once failed.

This isn't a sign of health — it's evidence that the check is too weak to fail. No matter what tX does, it returns 0 and prints OK, and the ledger faithfully records it as a pass. A gate that can't distinguish "the artifact works" from "the artifact returned zero" contributes exactly one thing to the ledger: a prettier average.

After excluding gX, the honest numbers are: lines drop from 1,852 to 1,665, passes drop from 579 to 392, and the pass rate drops from 31.3% to 23.5%. Four and a half percentage points of pass rate come from one trivial loop check.

The goals actually doing real work — those with 20+ distinct tasks — have pass rates between 0.0% and 29.8%. One of those goals ran 63 times with zero passes.

Why aggregate pass rates skew systematically high: When a ledger computes pass rate by line, every line carries equal weight, rather than being weighted equally by goal or task. This means that if a goal contributes a large number of records because it runs frequently and loops many times, it occupies a weight in the overall average far beyond its true value. gX's 187 lines are exactly this case: it has only one task, returns the same exit=0 OK every time, yet at the aggregate level it inflates the overall pass rate by roughly four and a half percentage points. This kind of "dilution effect" caused by low-frequency simple checks is common in monitoring and evaluation; the usual mitigation is to report stratified by goal, or to compute a pass rate for each goal separately before aggregating, so that high-frequency but information-free records don't dominate the conclusion.

What the Failures Actually Are

After grouping the 1,273 failures by the first line of their why field, a core problem of self-improving Agents is revealed:

Failure signature	Lines	Share of failures
exit=1 Traceback (most recent call last):	761	59.8%
JSON decode error	199	15.6%
exit=2 + uv/python path	96	7.5%
Regex not found	51	4.0%

The dominant failure mode is that when a self-improving Agent writes its own verification code, the code it generates throws an exception itself. It's not the task failing — it's the checker failing. The JSON decode error cluster is the same story one layer down: the Agent's own script printed non-JSON content, and then the downstream parser crashed.

These failures should not be deleted. A ledger you curate isn't a ledger, it's marketing.

Reading exit codes and the why field: In Unix-like and most script runtime environments, the process exit code carries a basic success/failure convention — 0 means success, 1 usually means a general error, 2 is often used for command-line usage errors, and higher values have meanings defined by specific tools. In OpenAmer's ledger, exit=1 followed immediately by Traceback indicates that the executed check script threw an uncaught exception; exit=2 accompanied by a uv/python path more often points to a call problem at the interpreter or dependency-environment level. These signals point not to "the artifact under test is defective" but to "the checker itself crashed." That's why preserving the raw output in the why field matters: only by seeing text like Traceback or a JSON decode error can you distinguish "task failure" from "verification infrastructure failure," avoiding the mistake of recording the fragility of the toolchain itself as artifact quality.

Lessons We Need to Remember

This audit yields two rules:

Report by goal, never just aggregate. The average across 223 goals can't tell you the real situation of any single goal. The by-goal table above is what should be looked at first.

Treat "≥20 lines, 100% pass" as suspicious rather than successful. Exactly one goal in this ledger matches that pattern. In a healthy population, that number should be zero, because real checks sometimes fail, and trivial checks get caught.

gX was not deleted. Deleting it would make the file prettier but less truthful. What it needs is a real assertion — that's follow-up work, not a rewrite of history.

The verification paradox of self-improving Agents: When an Agent is asked to write verification code for its own generated artifacts, a structural tension arises — the verifier is itself software generated by the same set of capabilities, and therefore inherits the same defect patterns. If the generated script has syntax, dependency, or output-format problems, it's usually the verifier that crashes first, not the artifact under verification, and so the failures piling up in the ledger reflect "the checker is unusable" more than "the product is substandard." To turn this noise into a usable signal, you typically need to decouple verification logic from generation logic: use relatively stable, independently tested checking tools, run health checks on the checkers themselves, or explicitly distinguish infrastructure failures from assertion failures in the ledger. Otherwise, a rise or fall in the failure rate may only indicate that the robustness of the checking code is fluctuating.

Verifying Yourself

• Format is one JSON object per line: {ts, task, goal, agent, passed, why}; why holds the raw check output. Everything above was computed from the live file by a ~20-line Python script.
• Public repo snapshot: docs/EVIDENCE-LEDGER-SNAPSHOT.json (615 results as of September 30, 2026).
• Everything runs CPU-only on a Windows laptop. No GPU, no cluster, no cloud inference.
• Repo: github.com/openamer/openamer (Apache-2.0 license).

The numbers here are a snapshot. The file grows every day, so the counts you read tomorrow will differ — that's the point of keeping a ledger instead of slides.

One-line Summary

An audit by OpenAmer of its own ledger found that 10.1% of pass records come from a weak check that has never failed; after excluding it, the true pass rate drops from 31.3% to 23.5%.

Paragraph Summary

The open-source AI Agent OpenAmer records every gate run in a line-by-line JSON ledger, writing exit codes and raw error text into the why field without self-scoring. Auditing 1,852 records revealed that the 31.3% aggregate pass rate is deceptive: the goal gX alone accounts for 187 lines, 100% passing, with just one task, whose why field is always exit=0 OK, making up 10.1% of the ledger. After excluding it, the true pass rate drops to 23.5%, and goals actually doing real work have pass rates of only 0.0% to 29.8%. Of the 1,273 failures, 59.8% are exceptions thrown by the Agent's own verification code rather than task failures. Two rules follow: report by goal rather than relying only on aggregates; treat "at least 20 lines and 100% pass" as a suspicious signal.