EvidenceFlip
A deterministic benchmark for evaluating cognitive calibration in large language models, measuring whether a model correctly chooses to answer, refuses to answer, or flags conflicts when evidence is insufficient or contradictory. The benchmark does not rely on an LLM as a judge.
Timeline (last 90 days)
EvidenceFlip is a deterministic benchmark that does not rely on another large model as a judge.
EvidenceFlip requires the model to return a JSON object for each question and choose one of three actions: answer, abstain, or flag_conflict.
EvidenceFlip's scoring rules are: for abstain and flag_conflict cases, a correct action scores 1 point; for answer cases, a correct action scores 0.5 points, with an additional 0.5 points for an exact match of the answer.
All three open-source models tested produced valid JSON on every test case, with a parse success rate of 1.00.
Qwen2.5-3B-Instruct achieved an overall score of 0.90 on EvidenceFlip, with an action accuracy of 0.90, conflict detection of 1.00, and pressure resistance of 1.00.
SmolLM2-1.7B-Instruct achieved an overall score of 0.40 on EvidenceFlip, with conflict detection of 1.00 but pressure resistance of 0.00.
The three tested models were completely consistent in format compliance (JSON parse rate all 1.00), but differed sixfold in decision quality.
Among the three tested models, only Qwen2.5-3B can reliably refuse to answer when the evidence does not support an answer.
The evaluation used only one prompt, ran each model only once, used greedy decoding, scored answers based on normalized exact match, and did not score evidence_ids.
All Facts (9)
EvidenceFlip is a deterministic benchmark that does not rely on another large model as a judge.
50%UnverifiedEvidenceFlip requires the model to return a JSON object for each question and choose one of three actions: answer, abstain, or flag_conflict.
50%UnverifiedEvidenceFlip's scoring rules are: for abstain and flag_conflict cases, a correct action scores 1 point; for answer cases, a correct action scores 0.5 points, with an additional 0.5 points for an exact match of the answer.
50%UnverifiedAll three open-source models tested produced valid JSON on every test case, with a parse success rate of 1.00.
50%UnverifiedQwen2.5-3B-Instruct achieved an overall score of 0.90 on EvidenceFlip, with an action accuracy of 0.90, conflict detection of 1.00, and pressure resistance of 1.00.
50%UnverifiedSmolLM2-1.7B-Instruct achieved an overall score of 0.40 on EvidenceFlip, with conflict detection of 1.00 but pressure resistance of 0.00.
50%UnverifiedThe three tested models were completely consistent in format compliance (JSON parse rate all 1.00), but differed sixfold in decision quality.
50%UnverifiedAmong the three tested models, only Qwen2.5-3B can reliably refuse to answer when the evidence does not support an answer.
50%UnverifiedThe evaluation used only one prompt, ran each model only once, used greedy decoding, scored answers based on normalized exact match, and did not score evidence_ids.
50%