Canary Audit
A validation mechanism that feeds outputs with known errors to the scorer before an evaluation run and requires it to reject all of them, used to ensure the scorer has not become ineffective due to changes.
Timeline (last 90 days)
Before any comparison run, feed outputs you know to be wrong into the scorer as canaries and require that all of them be rejected, to prevent the scorer from quietly starting to pass everything.
A canary that survives a scorer change should be treated as a stop-the-line event, not a warning.
All Facts (2)
A canary that survives a scorer change should be treated as a stop-the-line event, not a warning.
60%UnverifiedBefore any comparison run, feed outputs you know to be wrong into the scorer as canaries and require that all of them be rejected, to prevent the scorer from quietly starting to pass everything.
50%