Concept
Code Evaluator
An evaluation method that uses deterministic program logic to check whether an agent's output matches expectations. It is suited to structured scenarios with clearly defined rules, where programmatic checks can verify conformance to specified criteria. Because the checking logic is deterministic, results are reproducible and do not depend on subjective judgment. This approach is commonly used to validate outputs such as formatting, field values, and rule compliance in agent systems.
Timeline (last 90 days)
Oct 29
Deterministic checks (such as tool parameter schema or whether prohibited tools were called) should be handled by code evaluators, which are cheaper and fully reproducible.
Unverified50%