Unverified50% confidenceFactExact time
The author deliberately introduced a genuine severe regression that reduced an agent's success rate from 92% to 52%, yet the statistical rule originally designed for this purpose passed the PR because the regression was concentrated in a small number of tasks rather than being uniformly distributed.
1
Sources
50%
Confidence
Long-term
Relevance
9/27/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedAgents suffer from error accumulation in multi-step tasks; a ten-step sequential process where each step has 95% accuracy results in an overall success rate that drops to approximately 60%.74% similarUnverifiedAfter removing the candidate filtering component, FEVER accuracy dropped to 76.24%.69% similarUnverifiedIn a real-world case, a coding agent's success rate dropped from 92% to 88% without any changes, which was a false positive rather than a true capability regression.69% similarUnverifiedThe author admits that the 80% pass rate threshold felt somewhat arbitrary when first chosen, but has held up in practice.68% similarUnverifiedA benchmark score of 73.4% means that approximately one quarter of tasks still fall short, and this score was achieved under maximum compute.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/953019API
curl https://kongchang.com/api/v1/knowledge/claims/953019MCP
get_claim(id=953019)