Unverified50% confidenceFactExact time
SWE-bench Verified scores are significantly affected by the harness, and results from different implementations are not necessarily directly comparable.
1
Sources
50%
Confidence
Long-term
Relevance
10/24/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedIn code benchmarks such as SWE-bench, optimizations at the framework level have an even greater impact on final scores than differences in the models' own capabilities.80% similarUnverifiedSWE-bench scores are significantly influenced by the design of the outer agent scaffolding framework, and scores from different sources are not necessarily fully comparable.79% similarUnverifiedThe evaluation results show that the relationship between reasoning intensity and accuracy is not linear; at low intensity levels, some tasks exhibit obvious reasoning breaks, while the accuracy improvements brought by high intensity levels are often accompanied by a significant rise in token cost.77% similarUnverifiedWhen prior knowledge about possible fault patterns is available, selecting a sparse penalty structure that matches them often yields significant performance gains over general-purpose algorithms73% similarUnverifiedFor regression testing metrics, it is recommended to record exact match rate, key fact recall rate, refusal rate, and hallucination rate simultaneously; looking at multiple metrics together can avoid hidden degradation where the pass rate rises but answers become more conservative or more verbose.72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1094034API
curl https://kongchang.com/api/v1/knowledge/claims/1094034MCP
get_claim(id=1094034)