Unverified50% confidenceSolutionExact time
The evaluation adopted a paired-sample design in which each type of flaw had one flawed version and one structurally identical but problem-free twin version, used to measure the model's false-positive rate.
1
Sources
50%
Confidence
Long-term
Relevance
10/12/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedMcNemar 检验要求两个模型在完全相同的测试集上评估，任何测试集的不一致都会使检验失效78% similarUnverifiedAdversarial test cases should be written as deterministic binary pass/fail checkers rather than using fuzzy similarity scores.75% similarUnverifiedModels should be evaluated across multiple harnesses, multiple iterations, and multiple types of tasks, while distinguishing between model logic errors and toolchain issues.72% similarUnverifiedThe evaluation used only one prompt, ran each model only once, used greedy decoding, scored answers based on normalized exact match, and did not score evidence_ids.72% similarUnverifiedPassing the tests does not equate to correctly fixing the defect; models can cheat by deleting failing test cases, hardcoding expected output values, or modifying assertion logic to make it always evaluate to true72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1003791API
curl https://kongchang.com/api/v1/knowledge/claims/1003791MCP
get_claim(id=1003791)