Unverified50% confidenceDecision RuleExact time
In Agent scenarios, directly comparing the average success rates of two groups of runs is easily misleading, and a more robust approach is to use a paired experimental design.
1
Sources
50%
Confidence
Long-term
Relevance
10/16/2026
First Seen
Sources
Self-Evolving Agent Harness: The Self-Improvement of Agent Frameworks
hackernewshackernews10/16/2026
Related Entities
Related Claims
Unverified对同一测试用例执行多次调用统计成功比例得到通过率，比二元的通过/失败判断更能反映模型真实表现77% similarUnverifiedOn the strongest models, the differences between experimental groups fall within confidence intervals, and researchers characterized the results as directional patterns and soundness properties rather than proven performance lower bounds74% similarUnverifiedThe benchmark produces two numbers for each model that are never merged: a task score and a false-confidence rate (the proportion of cases that should have been escalated but were answered anyway).72% similarUnverifiedTwo models may be comparable in exact-match accuracy, but on shared test instances the number of cells they get wrong may differ greatly.72% similarVerifiedAgents suffer from error accumulation in multi-step tasks; a ten-step sequential process where each step has 95% accuracy results in an overall success rate that drops to approximately 60%.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1044693API
curl https://kongchang.com/api/v1/knowledge/claims/1044693MCP
get_claim(id=1044693)