Executable Benchmark
A benchmarking paradigm in which researchers provide a reference executor that can actually run the task, with the executor producing deterministic output. Model predictions are compared item by item with the executor's output, ensuring that evaluation is reproducible and auditable.
Timeline (last 90 days)
An executable benchmark requires researchers to provide a reference executor that can actually run the task, whose behavior is fully reproducible and auditable, avoiding the problem of ambiguity in manual annotation.
The executable benchmark paradigm is already fairly mature in fields such as code generation, SQL generation, and mathematical solving, but is still uncommon in spatial reasoning tasks.
All Facts (2)
An executable benchmark requires researchers to provide a reference executor that can actually run the task, whose behavior is fully reproducible and auditable, avoiding the problem of ambiguity in manual annotation.
50%UnverifiedThe executable benchmark paradigm is already fairly mature in fields such as code generation, SQL generation, and mathematical solving, but is still uncommon in spatial reasoning tasks.
50%