Evidence Sufficiency Evaluation
A benchmark designed to evaluate the evidence sensitivity of language models. It contains 72 test cases constructed using minimal pairs and adversarial control items, examining whether models can correctly identify scenarios involving insufficient, contradictory, or misleading evidence.
Timeline (last 90 days)
The Evidence Sufficiency Evaluation benchmark is designed to test whether language models can distinguish which answers are supported by existing evidence and which require reliance on unwarranted assumptions.
The benchmark contains 72 test cases.
The benchmark consists of 60 minimal-pair cases (30 minimal pairs spanning six categories) and 12 adversarial control cases.
The 12 adversarial control cases cover three types of scenarios: unresolved contradictions, unwarranted presuppositions, and near-miss distractors.
The benchmark uses expected labels, strict label parsing, and case-level correctness judgments, with the score reflecting the proportion of cases for which the model produces the expected label.
The authors state they will not provide unverified rankings and do not claim that any model comprehensively outperforms others, because a perfect score on 72 fixed cases cannot prove that a model performs equally well on unseen samples.
The benchmark has been made publicly available on Kaggle, and the authors welcome independent evaluations, the addition of more models, and feedback on the test design.
Benchmark results may be affected by prompt wording, model version, inference settings, and label parsing rules, and aggregate scores may obscure differences across categories.
All Facts (8)
Benchmark results may be affected by prompt wording, model version, inference settings, and label parsing rules, and aggregate scores may obscure differences across categories.
50%UnverifiedThe benchmark contains 72 test cases.
50%UnverifiedThe benchmark consists of 60 minimal-pair cases (30 minimal pairs spanning six categories) and 12 adversarial control cases.
50%UnverifiedThe 12 adversarial control cases cover three types of scenarios: unresolved contradictions, unwarranted presuppositions, and near-miss distractors.
50%UnverifiedThe benchmark uses expected labels, strict label parsing, and case-level correctness judgments, with the score reflecting the proportion of cases for which the model produces the expected label.
50%UnverifiedThe authors state they will not provide unverified rankings and do not claim that any model comprehensively outperforms others, because a perfect score on 72 fixed cases cannot prove that a model performs equally well on unseen samples.
50%UnverifiedThe Evidence Sufficiency Evaluation benchmark is designed to test whether language models can distinguish which answers are supported by existing evidence and which require reliance on unwarranted assumptions.
50%UnverifiedThe benchmark has been made publicly available on Kaggle, and the authors welcome independent evaluations, the addition of more models, and feedback on the test design.
50%