grader audit
A verification mechanism used in model evaluation that employs canary cases with known errors to test whether a grader incorrectly passes them, preventing the grader from silently failing after modification.
Timeline (last 90 days)
Before running comparisons, several known-bad outputs should be fed to the scorer and all required to be rejected, to audit whether the scorer passes everything because its rules have been loosened.
Scorer audits need two-way checks: require that wrong outputs be rejected and also that tolerance cases (near_miss) be correctly passed; a scorer that rejects everything creates a false regression story.
All Facts (2)
Before running comparisons, several known-bad outputs should be fed to the scorer and all required to be rejected, to audit whether the scorer passes everything because its rules have been loosened.
50%UnverifiedScorer audits need two-way checks: require that wrong outputs be rejected and also that tolerance cases (near_miss) be correctly passed; a scorer that rejects everything creates a false regression story.
50%