Reference Executor
In an executable benchmark, code or a program provided by researchers that can actually run the task and produce deterministic output. It is used for item-by-item comparison with model predictions, ensuring the objectivity and reproducibility of evaluation.
Timeline (last 90 days)
Grid-Bench actually runs operations through a reference executor, and every cell value output by the model can be checked item by item, making it a reproducible, auditable, precise computation task.
An executable benchmark requires researchers to provide a reference executor that can actually run the task, whose behavior is fully reproducible and auditable, avoiding the problem of ambiguity in manual annotation.
All Facts (2)
Grid-Bench actually runs operations through a reference executor, and every cell value output by the model can be checked item by item, making it a reproducible, auditable, precise computation task.
50%UnverifiedAn executable benchmark requires researchers to provide a reference executor that can actually run the task, whose behavior is fully reproducible and auditable, avoiding the problem of ambiguity in manual annotation.
50%