Unverified50% confidenceFactExact time
In this benchmark, each response contains two parts: a <think> block recording the model's stated reasoning process, and a <final> block specifying the action it intends to execute; a judge model first evaluates the final action, assigning labels such as SHORTCUT, HONEST_FAIL, and LEGIT_ESCALATION, with the reasoning trace serving as supporting evidence rather than the sole basis for judgment.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/12/2026
First Seen
Valid until: 1/10/2027
Sources
Related Entities
Related Claims
UnverifiedThe paper's conclusions are based on the model versions at the time of testing and represent a snapshot as of the test date, not a permanent judgment.75% similarUnverifiedThe evaluation uses an option-order reversal test (26 pairs) to distinguish whether a model truly understands the content or relies on option position to answer.73% similarUnverifiedWhen the true effect cannot be directly verified, a round of A/A testing can be run on candidate causal models first, and then the model to trust can be determined based on estimation error.73% similarUnverifiedThe evaluation program feeds the question, reference answer, and actual response together into the judge model, along with scoring instructions (e.g., relevance, accuracy, and completeness each scored 0–5), and the judge model outputs structured scores and reasoning.72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1007710API
curl https://kongchang.com/api/v1/knowledge/claims/1007710MCP
get_claim(id=1007710)