Unverified50% confidenceOpinionExact time
100 tasks can hardly cover the full complexity of wet labs, and benchmark performance does not equal zero errors in real experiments.
1
Sources
50%
Confidence
Long-term
Relevance
11/6/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedIn the experiment, the incident investigation and erasure request tickets achieved 100% completion across all configurations, while the seemingly simplest drop-column task proved the hardest.67% similarUnverifiedAmong the 95 tasks, only 14 tasks were truly solved by the best configuration (score no lower than 0.8 and no incidents).63% similarUnverifiedIn the first run, 50 Claude workers processed 19 tasks in parallel, all passed on the first attempt, the inbox remained empty, and local failures dropped from 139 to 0.63% similarUnverifiedThe developer ran 1932 test runs, using 100 realistic tools (including deliberately confusable tools such as pdf_compress, pdf_optimize, image_compress), with tool counts in 8 tiers from 5 to 100, running each condition twice, testing 18 tasks (including 10 ambiguous tasks).61% similarUnverifiedHuman researchers are constrained by experimental cost, time, and effort, limiting how many times they can peek at a test set, whereas automated research agents can run hundreds or thousands of experiments in a short period60% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1144333API
curl https://kongchang.com/api/v1/knowledge/claims/1144333MCP
get_claim(id=1144333)