Can Language Models Learn When to Refuse? Experiments with Policy Learning and Answer Selectors

An Underrated Capability: Knowing What You Don't Know

A truly reliable language model doesn't just answer questions correctly — it also needs to know when to say "I don't know." In factual question-answering scenarios, a model that fabricates an answer can do more harm than one that stays silent. Users get misled by false information, while an honest refusal actually builds trust.

This research centers on exactly that question: when should a model refuse to answer? The researchers tested two technical approaches on the Qwen3-14B model. The first involves training the model itself to learn when to respond. The second involves training a separate "answer selector" component that reviews the model's outputs and decides whether they're worth surfacing. Both approaches share the same core goal: making the optimal trade-off between correct answers, wrong answers, and abstentions.

Scoring Mechanism: Reward Correctness, Penalize Errors, Allow Silence

To evaluate the quality of the "answer or abstain" decision, the researchers designed a utility scoring framework. The logic is straightforward: correct answers earn points, wrong answers lose points, and choosing to abstain is a neutral, safe option. This design forces the model to weigh "risky answers" against "conservative silence," rather than simply optimizing for high coverage.

Experiments were run across three different training random seeds and two held-out confidence thresholds, to test the robustness of the conclusions and avoid any single-run artifacts. This multi-condition validation approach makes the results more convincing.

This utility scoring framework is technically known as the Selective Prediction paradigm. Its core idea is to formalize the model's output decision as a classification problem with a rejection option: given a question, the model can either produce an answer or emit an "abstain" signal. The utility function is typically written as $U = \alpha \cdot P(\text{correct}) - \beta \cdot P(\text{wrong})$, where $\alpha$ and $\beta$ represent the reward for a correct answer and the penalty for a wrong one, respectively. Adjusting these weights simulates different application risk profiles — in a medical context, for instance, where errors are costly, $\beta$ would be set much higher than $\alpha$, pushing the model toward conservative abstention. The confidence threshold serves as the execution mechanism: a model only outputs an answer when its confidence exceeds a preset threshold; otherwise, it defaults to abstaining. The threshold level directly governs the coverage-accuracy tradeoff curve.

Clear Improvement on TriviaQA

On the TriviaQA dataset, the learned answer selector achieved concrete improvements over the model's built-in calibrated confidence filter.

The numbers speak clearly: average coverage (the proportion of questions the model was willing to answer) rose from 20.2% to 30.9%, meaning the model was willing to tackle more questions. At the same time, the error rate among answered questions (under exact-match scoring) fell from 23.0% to 14.5%.

In other words, the selector helped the model answer more questions while getting fewer of them wrong — a rare double win. Crucially, a partially human-audited review validated this finding, ruling out any illusions caused by the scoring metric itself.

Reinforcement Learning Showed No Stable Gains

In contrast to the answer selector's success, the reinforcement learning (RL) modifications tested in this study did not yield consistent improvements. That is, directly training the model to internalize the abstention decision produced unstable results, with no reliable gains to speak of.

This result is telling in its own right. Rather than having a large model "internalize" the judgment of when to refuse, using an external, specialized selector to do that job may be a more practical and controllable engineering path. The model handles generation; the selector handles gatekeeping — and this separation of concerns actually delivers better outcomes.

RL's typical application in language models takes the form of RLHF (Reinforcement Learning from Human Feedback) or variants like GRPO and PPO. The core idea is to replace traditional supervised labels with reward signals, letting the model learn target behaviors through trial and error — in this study, that means learning autonomously when to answer and when to abstain. However, RL training is well known for high instability: sparse reward signals, policies prone to overfitting or collapse, and the phenomenon of "reward hacking," where the model finds shortcuts that earn high rewards without meeting the intended goal — for example, almost always abstaining to avoid penalty points. These characteristics mean that RL approaches in practice often require extensive tuning before converging stably, and their generalization is hard to guarantee. This study's results are consistent with common experience in the field, and help explain why the "dumber" but more controllable external selector approach won out.

NQ-Open Reveals a Harder Problem

The story doesn't end there, however. On the NQ-Open dataset, the answer selector still outperformed the confidence filter on threshold-averaged utility — but it performed worse than the "always abstain" baseline strategy.

This is a signal worth taking seriously. It suggests that in more difficult QA settings, the model struggles to identify questions that are even worth answering, to the point where answering nothing at all is the better choice. This exposes a core unresolved challenge: reliable abstention across datasets. A method that works on one dataset may simply fail on another.

TriviaQA and NQ-Open are two widely used open-domain QA benchmarks in natural language processing, but they differ substantially in difficulty distribution. TriviaQA questions come largely from competitive trivia, with relatively clear and unambiguous answers. NQ-Open (Natural Questions Open) comes from real Google search queries — questions that are more natural, more ambiguous, and in some cases have no single agreed-upon answer. This difference means that model accuracy on NQ-Open is generally lower than on TriviaQA, making the boundary of "which questions are worth answering" far less clear, and leaving the selector without a crisp decision rule to learn. This phenomenon in selective prediction research is called distribution shift — a selection policy trained on one dataset's assumptions doesn't transfer well to another dataset's difficulty distribution, and generalization naturally suffers.

Conclusions and Takeaways

This research delivers a careful, clear verdict: in factual QA settings, training an independent answer selector to decide when to respond is a promising technical direction, one that has already received support from human audits on TriviaQA. At the same time, it honestly acknowledges that getting a model to reliably abstain across different datasets remains an open problem.

For practitioners focused on AI reliability, a few takeaways are worth keeping in mind:

• Abstention capability should be treated as part of model quality, not an add-on feature;
• An external selector may be more practical than directly fine-tuning the model;
• Any abstention approach must be validated across multiple datasets — success on a single dataset is not enough to generalize.

Sometimes, teaching a model to stay silent is just as important as teaching it to speak.