Unverified50% confidenceCautionExact time
Using test passage as an optimization target leads to a Goodhart's Law trap, where agents may achieve the metric by modifying test assertions, skipping failing cases, or special-casing the tests rather than genuinely solving the problem.
1
Sources
50%
Confidence
Long-term
Relevance
10/10/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedWhen a metric becomes a target, it ceases to be a good metric (Goodhart's Law); when the industry races around benchmarks such as MMLU, HumanEval, and MATH, these tests become over-optimized.71% similarVerifiedGoodhart's Law states that when a metric becomes a target in itself it is no longer a good metric, and introducing test sets into training data causes a model's exam scores to decouple from its true capabilities71% similarUnverifiedAfter guardrails are removed, the model's capabilities themselves do not disappear—only the response strategy changes—making it very difficult to provide a reliable risk rating based solely on black-box testing71% similarUnverifiedBenchmark-based capability comparisons are typically built on specific testing conditions and are difficult to equate directly with complex, dynamic real-world security confrontation scenarios70% similarUnverifiedThe effectiveness of red-teaming is highly dependent on the model's unawareness of the testing context; a model that sees through the evaluation scenario will cause results to reflect only its performance capability.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/995186API
curl https://kongchang.com/api/v1/knowledge/claims/995186MCP
get_claim(id=995186)