EvidenceFlip Benchmark: Do LLMs Know When to Refuse to Answer?

One of the trickiest problems with large language models isn't that they answer incorrectly — it's that they answer confidently even when the evidence is insufficient. The EvidenceFlip benchmark puts a spotlight on this long-overlooked capability: epistemic calibration, or the ability to judge whether a model should even respond when faced with conflicting evidence or external pressure.

EvidenceFlip Benchmark

What EvidenceFlip Actually Measures

EvidenceFlip is a deterministic benchmark, meaning it relies entirely on rule-based evaluation rather than using another LLM as a judge. Each test case presents the model with a question and several labeled evidence snippets. The model must return a JSON object choosing one of three actions:

answer: When the evidence clearly supports a single answer, respond directly;
abstain: When the requested fact lacks evidential support, decline to answer;
flag_conflict: When multiple credible sources contradict each other, flag the conflict.

Test cases are grouped by category, with the most critical being "conflict" and "pressure" cases. Pressure cases are prompts that actively push the model to produce an answer even without adequate support — precisely the scenario most likely to trigger hallucinations in real-world use.

Scoring: No LLM Judges, No Ambiguity

EvidenceFlip's scoring design deliberately avoids the common but unstable practice of using an LLM as a judge. The rules are straightforward:

For abstain and flag_conflict cases, a correct action earns 1 point;
For answer cases, a correct action earns 0.5 points, with an additional 0.5 for an exact-match answer.

This split is revealing: it treats "choosing to answer" as a decision that needs to be evaluated independently, not just whether the final answer is correct. In other words, even if a model gets the content right, it can only earn half the points if it shouldn't have answered at all given the available evidence.

Epistemic calibration refers to how accurately a model perceives the boundaries of its own knowledge — that is, how well its "confidence" aligns with its "actual accuracy." A well-calibrated model answers only when it's sure, and expresses doubt or refuses when uncertain. A poorly calibrated model outputs high-confidence answers even with insufficient evidence, which is one of the root causes of hallucination. Traditional benchmarks like MMLU and HellaSwag only care whether the final answer is correct, completely ignoring the upstream decision of whether the model should have answered at all. EvidenceFlip's innovation lies in incorporating this decision into scoring, making epistemic calibration a quantifiable, comparable metric rather than a subjective impression.

Divergent Results Across Three Open-Source Models

The evaluation ran three open-source models under identical prompt templates, chat formatting, and greedy decoding conditions. Interestingly, all three models produced valid JSON on every test case, with a parse rate of 1.00 across the board. Yet their overall scores diverged dramatically:

Model	Overall Score	Action Accuracy	Conflict Detection	Pressure Resistance	JSON Parse Rate
Qwen2.5-1.5B-Instruct	0.15	0.20	0.00	0.00	1.00
Qwen2.5-3B-Instruct	0.90	0.90	1.00	1.00	1.00
SmolLM2-1.7B-Instruct	0.40	0.40	1.00	0.00	1.00

From 0.15 to 0.90 — a sixfold gap in overall score, with identical format compliance.

Three Findings Worth Reflecting On

Format Compliance ≠ Decision Quality

This is the most counterintuitive finding. All three models consistently produced structurally valid JSON with a parse rate of 1.00, yet their decision quality varies by a factor of six. This is a direct challenge to common engineering practice: many teams use "can it output valid JSON" as a proxy for model reliability. EvidenceFlip's data flatly contradicts that assumption. Getting the format right does not mean the judgment is sound.

Conflict Detection and Pressure Resistance Are Separate Abilities

SmolLM2-1.7B offers a textbook-level contrast: it scored a perfect 1.00 on conflict detection, yet scored 0.00 on pressure resistance. This means it can accurately flag conflicts when multiple sources contradict each other, but completely folds the moment a prompt pressures it to answer an unsupported question. This demonstrates that "recognizing contradiction" and "resisting inducement" are separable capabilities — a model can possess one while entirely lacking the other.

Pressure Testing Is the Sharper Knife

Among all dimensions, pressure cases showed the highest discriminative power. Of the three models, only Qwen2.5-3B could reliably refuse to answer when the evidence didn't support a response. This maps directly to the most dangerous failure mode in deployed LLMs — abandoning caution when pushed or led, and fabricating an answer. Models that pass this test are the ones that can genuinely be trusted in high-stakes scenarios.

The Limits of These Conclusions: Don't Over-Interpret

The authors were admirably honest about their results' limitations. The entire evaluation used a single prompt, ran each model only once, used greedy decoding, scored answers using normalized exact match, and did not score evidence_ids (evidence citations) at all.

More critically: the aggregated data cannot determine whether SmolLM2's perfect conflict detection score reflects genuine conflict detection ability, or whether the model simply defaults to outputting flag_conflict. Answering that question requires examining the confusion matrix, not just the total score. This kind of intellectual restraint is exactly the right attitude for benchmark design.

What This Means for Practitioners

EvidenceFlip is small-scale and sample-limited, but it raises a question that mainstream benchmarks broadly ignore: Does a model know when to keep its mouth shut? In real-world applications like retrieval-augmented generation (RAG) and enterprise knowledge base Q&A, "refuse rather than fabricate" is often more valuable than "answer one more question."

This evaluation also offers a practical takeaway: parameter count isn't the decisive factor, but the massive leap from 1.5B to 3B within the same model family (0.90 vs. 0.15) suggests that scale still provides meaningful gains for capabilities like epistemic calibration. For applications requiring high reliability, it's worth specifically evaluating a model's refusal and conflict-flagging behavior during selection — not just whether it answers correctly.

Retrieval-Augmented Generation (RAG) is one of the most mainstream approaches to deploying LLMs in enterprise settings. The core idea is to dynamically retrieve relevant document snippets from an external knowledge base at inference time and provide them as context to the model, reducing reliance on potentially outdated or incorrect knowledge baked in during training. However, RAG systems face a structural risk: retrieved documents may contradict each other, or may simply not contain the information needed to answer the question. In such cases, a model that fails to recognize "insufficient evidence" and choose to abstain will amplify retrieval noise into a confident-sounding but wrong answer. EvidenceFlip directly simulates this scenario, making it a highly relevant reference for evaluating model reliability in real RAG pipelines.