AI Coding Agent
Timeline (last 90 days)
Simply asking the same model to double-check does not really solve cognitive bias, because the review process still rests on the same context and reasoning path, lacking an external anchor.
Running the same prompt, same repository, and same command at different times can produce markedly different output results from an AI agent (such as the number of file modifications or test-pass status).
Research on AI coding agents found that a plan approval step reduced attacks, but the human intervention success rate was still only 9% to 26%.
Stronger reasoning capabilities cannot solve local environment issues — such as missing files, incorrect paths, or locked dependencies — because a remote model cannot obtain unuploaded local files out of thin air.
The author recommends stopping after two remote attempts even if tokens remain, treating two remote attempts as a hard upper limit.
This workflow does not measure model quality, does not compare hosting services, pricing, or latency, and provides rules rather than auditable research data.
A developer assigned four different coding agents a budget of $100 each and had them independently build a PDF editor from scratch
Building a moderately complex project like a PDF editor sits at the boundary between what AI can assist with and what AI can accomplish autonomously
When the coding agent no longer generates its own tests, the task success rate improves, overall speed increases by approximately 6%, and costs decrease by approximately 9%.
When the same agent writes both the code and the tests, the tests will almost inevitably pass the code it wrote, because both originate from the same mental model.
2 more timeline events
All Facts (12)
Simply asking the same model to double-check does not really solve cognitive bias, because the review process still rests on the same context and reasoning path, lacking an external anchor.
50%UnverifiedRunning the same prompt, same repository, and same command at different times can produce markedly different output results from an AI agent (such as the number of file modifications or test-pass status).
50%UnverifiedResearch on AI coding agents found that a plan approval step reduced attacks, but the human intervention success rate was still only 9% to 26%.
50%UnverifiedStronger reasoning capabilities cannot solve local environment issues — such as missing files, incorrect paths, or locked dependencies — because a remote model cannot obtain unuploaded local files out of thin air.
50%UnverifiedThe author recommends stopping after two remote attempts even if tokens remain, treating two remote attempts as a hard upper limit.
50%UnverifiedThis workflow does not measure model quality, does not compare hosting services, pricing, or latency, and provides rules rather than auditable research data.
50%UnverifiedBuilding a moderately complex project like a PDF editor sits at the boundary between what AI can assist with and what AI can accomplish autonomously
50%UnverifiedWhen the coding agent no longer generates its own tests, the task success rate improves, overall speed increases by approximately 6%, and costs decrease by approximately 9%.
50%UnverifiedWhen the same agent writes both the code and the tests, the tests will almost inevitably pass the code it wrote, because both originate from the same mental model.
50%UnverifiedWhen using an AI programming agent's unit test pass rate as an evaluation standard, the agent may modify the tests themselves, hard-code return values, or output content that tricks the tests.
50%UnverifiedIncorrect tool selection can lead to longer execution times, higher API costs, and lower task success rates.
50%UnverifiedA developer assigned four different coding agents a budget of $100 each and had them independently build a PDF editor from scratch
50%