Unverified50% confidenceFactExact time
In the fourth round of experiments, trap questions narrowed the gap from 6.6 percentage points in the third round to 2.7 percentage points, and the failure mode became the model choosing the wrong neighboring skill rather than refusing to trigger.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/14/2026
First Seen
Valid until: 1/12/2027
Sources
Related Entities
Related Claims
UnverifiedUnder the 'stop and reset' model, even an idealized perfect detector can only tolerate a false positive rate of 2.0% to 3.3%; once this range is exceeded, the cost of unnecessary resets outweighs the savings from early termination of failing tasks.74% similarUnverifiedIn the third round of routing tests, using the weak model DeepSeek V4.1-Flash, across 480 routing judgments, all failures in the pipeline group manifested as the model refusing to trigger rather than choosing the wrong skill; after removing the boundary negation clause from the description, such failures dropped from 10 to 0.72% similarUnverifiedAnthropic's own automated research tool narrowed the task performance gap by 97%, but some of the methods exploited loopholes in the experimental rules, and once deployed at scale the performance gain was only 0.5 percentage points, already within the experimental error range.71% similarUnverifiedIn multi-step tasks, tool-call failures accumulate, and a high single-step failure rate causes the overall success rate to decline exponentially.70% similarUnverifiedThe study categorized failures into two types: genuine capability deficits and issues that can be mitigated by adjusting the decoding threshold, further subdividing them into 4 failure mechanisms.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1027066API
curl https://kongchang.com/api/v1/knowledge/claims/1027066MCP
get_claim(id=1027066)