Expired50% confidenceBenchmarkExact time
Across 18 runs of the two-bug repair task (three per setting), the model passed all 215 hidden tests perfectly regardless of effort setting, with no measurable difference in quality.
1
Sources
50%
Confidence
Short-term (~14 days)
Relevance
10/19/2026
First Seen
This claim has expired. It is kept for the record and no longer describes the current state.
Valid until: 11/2/2026(expired)
Sources
Related Entities
Related Claims
UnverifiedReproducible tests should freeze a test set of approximately 20 real tasks, record the hardware and runtime environment, and use blind scoring with model names hidden.72% similarUnverifiedThe evaluation adopted a paired-sample design in which each type of flaw had one flawed version and one structurally identical but problem-free twin version, used to measure the model's false-positive rate.71% similarUnverifiedSoloBooks AI's test suite runs on Vitest, with all 17 tests passing, covering minor-unit arithmetic, document security, schema validation, duplicate detection, invoice status transitions, and real Ollama connectivity.70% similarUnverifiedThe test suite passed on its first run: 17 gate tests covering mutual exclusion, busy skipping, fail-open, and cross-process flock scenarios, plus 74 fallback and turn lifecycle regression tests.69% similarUnverifiedIn the demo, the agent first ran the tests to locate the problem with two failing tests, re-ran them after fixing, and ultimately all five tests passed, with a three-point summary.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1067175API
curl https://kongchang.com/api/v1/knowledge/claims/1067175MCP
get_claim(id=1067175)