HarnessDev Paper Explained: Feedback and Held-out Scores Agree Only 53%, Self-Improvement Loops Suffer from Selection Bias

A paper recently discussed in the Reddit community, HarnessDev, measures a process that few had previously quantified: letting an AI agent modify its own execution harness, deciding whether to keep each change based on feedback, and looping this process. Narratives of this kind of "self-improvement" are hardly rare in AI circles, but HarnessDev is arguably the first to lay out and quantify the agreement between feedback signals and held-out signals.

Why Feedback Scores Are Unreliable

The paper's core data is blunt. Across 64 adjacent version switches, feedback scores and held-out scores moved in the same direction only 34 times, an agreement rate of 53.1%. In other words, if you rely solely on the feedback signal to judge whether a harness modification is actually an improvement, your accuracy is roughly that of a coin flip.

What is more concerning is the final result: of 9 evolutionary lineages, only 2 ended up on the version that performed best on the held-out set. The average gain during the evolutionary phase was +3.11 points, but whether that gain generalizes is something the feedback gate itself cannot reliably answer.

This points to a long-overlooked issue — the selection policy. The paper's authors and the Reddit poster reach the same conclusion: this is not a problem of the proposer being unable to generate good candidates, but a problem of the selection policy. Even if the agent can propose good modifications, if the filtering mechanism itself is biased, good modifications may be rejected while bad ones are retained.

The distinction between feedback score and held-out score is the key premise for understanding the entire paper. The feedback score refers to the evaluation signal the agent actually receives during the evolution loop — typically drawn from the set of tasks it is optimizing, tasks that are visible to the proposer and can be revisited repeatedly. The held-out score is measured on a separate set of tasks that the proposer has never seen and that did not participate in selection, with the goal of testing whether the modification truly generalizes. Low agreement between the two is essentially a concrete manifestation of the overfitting-versus-generalization gap in the agent self-improvement setting: the agent can push scores higher on visible tasks, but those gains don't necessarily transfer to unseen tasks. The classic train/validation/test split in machine learning exists to prevent exactly this kind of leakage, and HarnessDev's contribution is to quantify how severe that leakage is when the agent is both proposer and evaluation subject — 53.1% means the feedback signal carries almost no useful information about generalization.

Executor Dependency: The Same Harness Performs Dramatically Differently Across Models

The paper also reveals a troubling phenomenon: executor dependency is extremely severe. The same harness scores 69.3 under one model and only 33.0 under another. This means harness evaluation results are highly dependent on which base model they run on, and cross-model transferability is far worse than one might imagine.

This finding has direct implications for real deployments. If your evaluation pipeline uses only one model to compute feedback scores, those scores will likely become completely invalid once you switch models. And the 53.1% agreement rate in the paper is, to some extent, a side reflection of this dependency.

Executor here refers to the base model that carries the agent, while the harness is the execution framework wrapping the model — including prompt templates, tool-calling protocols, loop control, result parsing, and so on. Executor dependency means the benefits of a harness are not model-agnostic: a tool-calling format that works on GPT-4 may fail entirely on a model with weaker capabilities or a different instruction-following style. This coupling has direct engineering consequences. First, it makes harness evaluation results non-portable across models; a framework tuned today may reset to zero after a model switch. Second, it makes the "self-improvement" loop more fragile — if the underlying model is upgraded or downgraded during evolution, previously accumulated harness modifications may all become invalid at once. The drop from 69.3 to 33.0 in the paper suggests that when evaluating a harness you must report the full model–harness combination, rather than claiming in isolation that a given harness is "better."

Improvement Ideas: Rejection Memory and Conservative Release Strategies

The poster proposes two improvement directions distinct from the paper's loop and implements these strategies in the tool reef that they maintain.

Release Threshold: From Net Wins to No Regression

The default policy is: allow release if the candidate wins more task comparisons than it loses and exceeds a set noise floor. The Pareto policy in the methodology guide is stricter — release only when no task regresses and at least one task improves.

But there is a subtle trap here: this rule can only reject candidates on the lists where the gate runs, and once you make selections on a given list, that list becomes your feedback score rather than your held-out score. So on its own it cannot recover the losses from the 30 bad switches in the paper. What it can do is: prevent a candidate that lost somewhere on the list from being released solely on the strength of net wins. The held-out check must use a list the proposer has never seen, and this must be guaranteed by the user.

The Pareto policy borrows the concept of Pareto improvement from multi-objective optimization: a candidate is accepted only if it makes no individual task worse and at least one task better. Compared with a "net positive wins" rule, Pareto excludes candidates that "win more than they lose but contain clear regressions" — the latter look good on average but may introduce severe degradation on some tasks. The cost is an extremely low acceptance rate, because genuine improvements often come with local trade-offs. This is also the technical root of the open question in the paper: the Pareto rule is inherently conservative and can filter out false positives when signals are noisy, but it may also reject genuine yet non-uniform gains because its requirements are too strict. Understanding this is necessary to see that "no regression" is not merely risk aversion, but a statistical defense strategy when the reliability of the feedback signal is only 53%.

Rejection Memory: Stop the Proposer from Repeating Failures

The proposer can receive recently rejected proposals along with the reasons for their rejection, as well as the failure list from the previous step. This allows it to avoid repeatedly proposing the same switch that has already been vetoed. This is a low-cost improvement, but the paper's loop does not appear to include such a mechanism.

The proposer is the component in the self-improvement loop responsible for generating candidate modifications, usually driven by an LLM, which proposes the next change based on the current harness state and historical feedback. "Rejection memory" is a form of context engineering: feeding previously vetoed proposals and the reasons for their rejection back to the proposer as additional context, so that it no longer repeats known-failed paths. This is conceptually related to negative experience replay in reinforcement learning or regression testing in software engineering, but its implementation cost is extremely low — you only need to attach a rejection log to the prompt. Its limitation lies in the finite context window; after long-term evolution the log will expand, requiring summarization or retrieval mechanisms to compress it. In addition, it can only avoid repeating mistakes, not directly guide the proposer toward new directions. The absence of this mechanism in the paper's loop means the agent may repeatedly probe the same dead end, wasting the evolution budget.

The Task List Is the Fundamental Bottleneck

Finally, the poster points out a problem that no improvement can bypass: the task list itself.

The list is short because every step has a cost. But the entire paper is precisely warning that a short, in-distribution task list will gradually be "gamed" by the feedback loop. The agent will learn to farm points on this specific list rather than genuinely improving its capabilities.

This leads to an open question: is using no regression on held-out tasks as the acceptance rule the right approach, or is it so conservative that progress can never be made? The data on those 30 bad switches in the paper makes the poster lean toward conservatism. With feedback signal reliability at only 53%, conservatism may be the more rational choice — better to miss some real gains than to let regressions slip in quietly.

The answer to this question may depend on the specific scenario: if your harness evolution cycle is short and the cost of human review is low, an aggressive strategy may be viable; but for long-running unattended automatic evolution, a conservative acceptance rule is almost the only reasonable choice.

One-line Summary

HarnessDev is the first to quantify the loop in which an agent self-improves its harness, finding only 53.1% agreement in feedback signals, with the bottleneck lying in the selection policy.

Paragraph Summary

HarnessDev is the first paper to systematically measure the loop in which an AI agent modifies its own execution harness. Across 64 adjacent version switches, feedback scores and held-out scores agreed only 53.1% of the time, meaning that judging whether a modification is good based solely on the feedback signal is no better than a coin flip. Only 2 of 9 evolutionary lineages ended up on the best held-out version, leading the authors to conclude that the problem lies in the selection policy rather than proposer capability. The paper also finds severe executor dependency, with the same harness scoring anywhere from 69.3 to 33.0 depending on the model, indicating poor cross-model transferability. The Reddit poster proposes two improvements: using Pareto no-regression as the release threshold and giving the proposer rejection memory. But the fundamental bottleneck is that the task list is too short and in-distribution, making it susceptible to being gamed by the feedback loop. At 53% signal reliability, whether a conservative acceptance rule is too conservative remains an open question.