HarnessDev
A research paper quantifying the consistency between feedback signals and held-out signals in AI agent self-improvement loops. It finds that the agreement rate between the two is only 53.1%, and that harnesses exhibit severe executor dependency issues.
Timeline (last 90 days)
The HarnessDev paper measured the loop in which an AI agent modifies its own execution harness and decides based on feedback whether to keep the modification, and for the first time quantified the consistency between the feedback signal and the held-out signal.
Among 64 adjacent version switches, the feedback score and the held-out score moved in the same direction only 34 times, a consistency rate of 53.1%.
Among 9 evolutionary lineages, only 2 ultimately settled on the version with the best performance on the held-out set.
The average gain during the evolution phase was +3.11 points, but whether that gain generalizes is a question the feedback gate cannot answer reliably.
The paper's authors and the Reddit poster judged that HarnessDev's problem lies in the selection policy rather than insufficient capability of the proposer.
The same harness scored 69.3 under one model but only 33.0 under another, indicating an extremely severe dependence on the executor.
The rejection memory mechanism lets the proposer receive the most recently rejected proposals and the reasons for their rejection, as well as the failure list from the previous step, so as to avoid repeatedly proposing switches that have already been rejected; the paper's loop apparently did not include this mechanism.
All Facts (7)
The HarnessDev paper measured the loop in which an AI agent modifies its own execution harness and decides based on feedback whether to keep the modification, and for the first time quantified the consistency between the feedback signal and the held-out signal.
50%UnverifiedAmong 64 adjacent version switches, the feedback score and the held-out score moved in the same direction only 34 times, a consistency rate of 53.1%.
50%UnverifiedAmong 9 evolutionary lineages, only 2 ultimately settled on the version with the best performance on the held-out set.
50%UnverifiedThe average gain during the evolution phase was +3.11 points, but whether that gain generalizes is a question the feedback gate cannot answer reliably.
50%UnverifiedThe paper's authors and the Reddit poster judged that HarnessDev's problem lies in the selection policy rather than insufficient capability of the proposer.
50%UnverifiedThe same harness scored 69.3 under one model but only 33.0 under another, indicating an extremely severe dependence on the executor.
50%UnverifiedThe rejection memory mechanism lets the proposer receive the most recently rejected proposals and the reasons for their rejection, as well as the failure list from the previous step, so as to avoid repeatedly proposing switches that have already been rejected; the paper's loop apparently did not include this mechanism.
50%