Unverified50% confidenceBenchmarkExact time
Khoj's benchmark injects 200 new projects into a copy of the graph, half hiding one of four risk patterns and half being clean controls with traps; the ablation without the two structural rules found almost all risks but mislabeled 93% of the clean projects.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/18/2026
First Seen
Valid until: 1/16/2027
Sources
Related Entities
Related Claims
UnverifiedThe first version of the task set was 100% trick questions, with a single-attempt baseline of 0/12, making it impossible to measure the harm of ungated correction; the harm only became measurable after adding clean tasks.69% similarUnverified对抗测试的效果高度依赖场景库的完备性，未被预见的攻击路径依然是盲区，单次通过测试不代表Agent在生产中永远安全66% similarUnverifiedDuring the orchestration experiment, the entire build process had approximately 30 substantive defects, which were resolved across four rounds of fixes with none remaining in the final version, all of which were discovered through external validation and never proactively detected by the model itself.65% similarUnverifiedThe project's observations found: the gap between the notice rate and the safe action rate is far larger than expected -- a model can flag a risk in its reasoning trajectory but still issue a dangerous tool call two steps later, because there is no mechanism in the scaffolding to enforce a hard stop when a risk is detected. This is a system design gap, not just a model capability gap.64% similarUnverifiedThe core conclusion of the experiment is: a step that re-examines the original material rather than trusting the previous step's summary can catch real errors; whereas a planning step produced through an overly narrow schema handoff buys no reliable contribution.63% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1062682API
curl https://kongchang.com/api/v1/knowledge/claims/1062682MCP
get_claim(id=1062682)