A Reader Took Apart My Benchmark: Data Corrections and Reflections on the Open-Source Tool Receipts

When the Reader Is More Careful Than the Author

Last week, I published a study of 181 real code changes. One sentence in it was wrong. A reader caught it by re-adding my numbers. This post will tell you: what they found, what I changed, and what's still unresolved.

The episode itself isn't flattering, but it offers a rare sample — what happens when the author of a benchmark is willing to be publicly corrected. More valuable still, the correction process exposed a statistical phenomenon more interesting than the original conclusion.

What That Sentence Got Wrong

In my original text, I wrote: in 10% of agent-authored pull requests (PRs), all tests fail on the old code, for the sole reason that the test file imports a name added by the change (so the file can't even load), and "in my sample, no maintainer commit looks like this."

Before reading the post, the reader re-derived my result files. All the sums held. Then they pointed to a row I'd skipped: the maintainer file listed 9 of 212 tests as WEAK (4%), while the agent file listed 79 of 591 (13%). So the pattern isn't absent on the maintainer side.

I went and looked. All nine come from a single marshmallow feature commit that added a new keyword argument. Its test file loads fine, and the same commit also contains tests that genuinely prove the change. So that sentence was originally about "every test is WEAK," and the fix is to describe honestly what's actually in the maintainer data. Counted by change, 14 of 91 agent PRs contain a WEAK test, versus just 1 of 71 maintainer commits.

WEAK / PROVEN, which recurs throughout the post, are the test-quality tiers the author defined for this study. PROVEN means the test fails on the old code and passes on the new code — i.e., it genuinely "proves" the change is necessary; WEAK means the test exists, but its failure reason doesn't point to the behavior under test — for example, the test file can't even load because it imports a name that doesn't yet exist, so the test dies during collection or import rather than at an assertion. The distinction matters: a WEAK test appears to cover the change but actually verifies no logic. The change-level rule tends to exclude changes whose overall verdict is WEAK from being PROVEN, which is why the reader pointed out that "part of the split is a readback of the aggregation rule."

After the Split, It Gets More Interesting

The reader's next question: does the import pattern only show up when nothing else proves the change? Or does it also appear when there's already a proving test?

Splitting the 91 agent PRs by whether they contain a proving test:

• With a proving test: 5 of 80 carry a WEAK test (6.3%, 95% CI 2.7 to 13.8).
• Without a proving test: 9 of 11 (81.8%, interval 52.3 to 94.9).

If WEAK tests appeared at the same 6.3% rate everywhere, you'd expect about 0.7 of 11 to carry one. The actual number is 9. So the bulk lands exactly where nothing else proves the change.

The reader also raised a key issue: my change-level rule never calls a change that carries a WEAK test Proven, so part of this split is really a readback of the aggregation rule. The 5 of 80 is the only figure immune to this, because those PRs both provide proof and add a WEAK test. It's a writing rate — small, but not zero.

I personally checked their statistics (Wilson intervals, Fisher's exact test). They hold. The maintainer side is a different matter: 1 of 71 is a single event, and no matter how clean the contrast looks, it can't tell anyone about a base rate.

A Wilson confidence interval is a way to construct an interval estimate for a binomial proportion without assuming a normal approximation, making it more robust than the common Wald interval when sample sizes are small or proportions are near 0 or 1. The "5/80, 95% CI 2.7 to 13.8" in this post is computed with that method: it tells us that if we repeated the same sampling process many times, the true proportion would fall within this range roughly 95% of the time. The smaller the sample, the wider the interval — which is exactly why "1 of 71" on the maintainer side can't support any base-rate conclusion.

Fisher's exact test is used to determine whether two binary variables are independent — for example, whether "carries a WEAK test" and "has a proving test" are associated. It computes exact probabilities directly from the hypergeometric distribution without relying on large-sample approximations, making it well suited to the contingency tables with very small cell counts in this post. The reader used it to test whether the difference between 6.3% and 81.8% could be due to random fluctuation alone; the result supports a real difference.

What Specifically Changed

• The study document now states all of the above, including the intervals, and uses the same "weak-only" definition everywhere (previously my generator and two result files disagreed with the doc).
• The tool Receipts now prints how each PROVEN test failed on the old code. Another Reddit reader said "it went red" isn't enough: you want to see that the test reached an assertion, not that it died on an import. The tool knew this but wasn't showing it.
• I wrote down the next step before running it: collect a few hundred more maintainer commits, as their own row, and apply the same selection rule to both sides. The reader predicts the maintainer rate will move toward the agent rate, since that one maintainer event is itself a churn-shaped commit. This prediction is now recorded in the doc, so it will be interpreted against it whichever way the result goes.

The author's mention that "the generator and two result files disagreed with the doc" touches on a common problem in data pipelines: the single source of truth. When a definition (here, the "weak-only" criterion) is scattered across a generator script, multiple result files, and documentation, any change in one place can cause the criterion to drift, ultimately making the numbers in the prose inconsistent with the reproducible results. Pulling the definition back into one place and having the tool read from the same source is a prerequisite for making a benchmark auditable and reproducible. This is also why the reader was able to "re-derive the result files" first and then pinpoint the specific row of data.

Two Things I Took Away

First: before arguing with someone's sentence, re-derive their sums — because the sentence is exactly where things get loose. Second: a nice headline number ("10% of agent PRs") can be both a fact about your decision rule and a fact about the code, and the only way to discover that is a reader with no stake in the outcome.

The tool is open-source and free: https://github.com/syntaxixr/receipts. Every change in the study document is linked: https://github.com/syntaxixr/receipts/blob/main/docs/study.md.

Note: I drafted this post with an AI assistant. The numbers come from the study files in the repository and the comment thread under the original post.

One-line Summary

A reader re-checked the author's benchmark, caught one wrong conclusion, and revealed a more valuable statistical finding.

Paragraph Summary

The author published a study of 181 real code changes, in which the claim "no maintainer commit looks like this" was overturned by a reader who re-added the data: WEAK tests also exist on the maintainer side. Splitting by change shows 81.8% of WEAK tests appear in agent PRs with no other proving test, versus just 6.3% when a proving test exists; the difference holds under Wilson intervals and Fisher's exact test. The author acknowledges the aggregation rule itself read back part of the conclusion, and accordingly revised the doc, unified the "weak-only" definition, upgraded the Receipts tool to show how each PROVEN test specifically fails on old code, and pre-registered a prediction about extending to maintainer commits.