Unverified50% confidenceSolutionExact time
Before any comparison run, feed outputs you know to be wrong into the scorer as canaries and require that all of them be rejected, to prevent the scorer from quietly starting to pass everything.
1
Sources
50%
Confidence
Long-term
Relevance
10/19/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedScorer audits need two-way checks: require that wrong outputs be rejected and also that tolerance cases (near_miss) be correctly passed; a scorer that rejects everything creates a false regression story.75% similarUnverifiedFlaws in the evaluator interface can masquerade as reasoning failures, and failing to distinguish between the evaluation pipeline and model capability when interpreting benchmark results may lead to misleading conclusions.72% similarUnverifiedImprovements manufactured by observing the evaluation set to guide selection, cherry-picking the best runs, or reporting only winning results will vanish on new data.71% similarUnverifiedA skip request is rejected if any field is missing, if the results of two runs are consistent, or if the basis for the determination has been modified by the same patch69% similarUnverifiedThe first-run scorer only accepted lines beginning with `ANSWER:`, causing most answers judged as incorrect to actually be correct—for example, correct responses with the number and label on the same line, Markdown bold formatting, or LaTeX format were all scored 0.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1066532API
curl https://kongchang.com/api/v1/knowledge/claims/1066532MCP
get_claim(id=1066532)