Unverified50% confidenceFactExact time
The test repository chosen was python-humanize/humanize (MIT license, 1725 lines of Python, about 860 tests run in 7 seconds), pinned to a specific commit. The four tasks were: a bug fix for a real issue, implementing a new feature to spec (16 hidden acceptance tests), a refactor with no behavior change, and a code review (3 planted bugs).
1
Sources
50%
Confidence
Long-term
Relevance
10/14/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedReproducible tests should freeze a test set of approximately 20 real tasks, record the hardware and runtime environment, and use blind scoring with model names hidden.67% similarVerifiedHumanEval是由OpenAI于2021年发布的代码生成基准测试，包含164个手写的Python编程问题66% similarUnverifiedHumanEval tests whether a model can complete isolated function snippets, whereas real-world engineering more commonly involves locating issues in large codebases, understanding business logic, and making cross-file modifications.64% similarUnverifiedAfter an agent claims completion, the Software Factory re-runs tests and has a dedicated reviewer agent inspect the code diff to catch issues that test coverage cannot detect.64% similarUnverifiedDuring testing, one should enter a clean test directory to perform small tasks, because some Agent-type tools may perform write operations on files within the directory during the testing phase, posing a risk of accidentally modifying code.64% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1025018API
curl https://kongchang.com/api/v1/knowledge/claims/1025018MCP
get_claim(id=1025018)