ClaimScope
A benchmark for evaluating the mathematical claim-auditing ability of large models. It contains 72 tasks requiring the model to output a verdict, scope of applicability, counterexamples, and a repair plan, and employs a label-blind witness verification mechanism.
Timeline (last 90 days)
ClaimScope is a benchmark containing 72 tasks, used to evaluate a model's ability to audit mathematical propositions, requiring the model to output a quadruple of verdict, scope of applicability, counterexample, and repair plan.
ClaimScope covers three types of mathematical problems: optimizer state transfer, finite-horizon affine recurrence, and exact 2×2 linear algebra.
ClaimScope's core mechanism is a label-blind witness checker, which only checks the state submitted by the model, does not refer to the verdict in the reference answer, and does not provide an alternative witness.
On invalid propositions, using verifier feedback yielded 23/39 accepted counterexamples, while generic feedback without a checker yielded only 15/39.
For safe completion, verifier feedback was 25/72, and generic feedback without a checker was 20/72.
Strict completion was 17/72 under both conditions, and verdict accuracy went from 47/72 to 46/72, remaining almost unchanged.
The safe-completion difference grouped by claim was .094, with a bootstrap 95% confidence interval of [.000, .198].
A single post-hoc pure schema correction applied to the direct output, without calling the model, can also achieve the same 25 safe tasks.
The ClaimScope paper characterizes the observed gain as witness-structure correction rather than a leap in the model's reasoning ability.
Schema-only post-hoc correction can achieve a number of safe completions comparable to verifier feedback; possible reasons are that the model's raw output already contains the correct content but in a messy format, or that verifier feedback mainly imposes format constraints rather than providing new reasoning signals.
1 more timeline events
All Facts (11)
What verifier feedback truly improves is the witness structure, not the ability to classify propositions.
50%UnverifiedClaimScope covers three types of mathematical problems: optimizer state transfer, finite-horizon affine recurrence, and exact 2×2 linear algebra.
50%UnverifiedClaimScope's core mechanism is a label-blind witness checker, which only checks the state submitted by the model, does not refer to the verdict in the reference answer, and does not provide an alternative witness.
50%UnverifiedOn invalid propositions, using verifier feedback yielded 23/39 accepted counterexamples, while generic feedback without a checker yielded only 15/39.
50%UnverifiedFor safe completion, verifier feedback was 25/72, and generic feedback without a checker was 20/72.
50%UnverifiedStrict completion was 17/72 under both conditions, and verdict accuracy went from 47/72 to 46/72, remaining almost unchanged.
50%UnverifiedThe safe-completion difference grouped by claim was .094, with a bootstrap 95% confidence interval of [.000, .198].
50%UnverifiedA single post-hoc pure schema correction applied to the direct output, without calling the model, can also achieve the same 25 safe tasks.
50%UnverifiedThe ClaimScope paper characterizes the observed gain as witness-structure correction rather than a leap in the model's reasoning ability.
50%UnverifiedSchema-only post-hoc correction can achieve a number of safe completions comparable to verifier feedback; possible reasons are that the model's raw output already contains the correct content but in a messy format, or that verifier feedback mainly imposes format constraints rather than providing new reasoning signals.
50%UnverifiedClaimScope is a benchmark containing 72 tasks, used to evaluate a model's ability to audit mathematical propositions, requiring the model to output a quadruple of verdict, scope of applicability, counterexample, and repair plan.
50%