Unverified50% confidenceDecision RuleExact time
The three practical principles summarized by the Sturdybench team are: every no-call case must come with a secondary reply check; reply word lists should choose words around the user's request rather than around the refusal; and failures should be reported by failure direction.
1
Sources
50%
Confidence
Long-term
Relevance
10/12/2026
First Seen
Sources
Related Entities
Related Claims
Unverifiedτ-bench's criterion is: even if the agent successfully helps the user refund a ticket, as long as the refund action violates policy, the task is still judged as failed, i.e., it measures "task completion under the premise of following the rules".67% similarUnverified'Unavailable' and 'not applicable' require different handling logic: the former is a temporary issue that can be retried, while the latter is a structural limitation requiring the user to adjust their request.67% similarUnverifiedRigorous third-party evaluation must attempt to reach the true upper limit of a model's capabilities rather than merely testing default behavior, because model performance is highly dependent on prompt design and may exhibit different capability boundaries under jailbreak prompts.66% similarUnverifiedThe prerequisite for this failure mode is that the policy must be precisely specified in the input, and it only holds when the rules are clearly visible yet the model still believes contradictory verbal assertions66% similarUnverifiedThe 'rule of three' advises that when a problem has not yielded a satisfactory result after three attempts, a new conversation should be started.66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1009593API
curl https://kongchang.com/api/v1/knowledge/claims/1009593MCP
get_claim(id=1009593)