Unverified50% confidenceFactExact time
SWE-bench scores depend heavily on the dependency versions, timeout settings, and evaluation scripts of the runtime environment, which explains why the same model can get vastly different results in the hands of different teams.
1
Sources
50%
Confidence
Long-term
Relevance
10/16/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedMost frontier models exhibit extremely high variance when running the same task repeatedly, potentially producing drastically different results.72% similarUnverifiedSWE-bench scores are significantly influenced by the design of the outer agent scaffolding framework, and scores from different sources are not necessarily fully comparable.72% similarUnverifiedIn code capability benchmarks such as SWE-bench, the performance gap for the same underlying model under different harnesses can reach tens of percentage points71% similarUnverifiedThe same model paired with different harnesses can exhibit vastly different success rates on real coding tasks.70% similarUnverified通用评测分数接近的大模型在特定垂直任务上可能表现差异巨大70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1049233API
curl https://kongchang.com/api/v1/knowledge/claims/1049233MCP
get_claim(id=1049233)