RLEC: Teaching LLMs to Decide When to Answer Based on Confidence Targets

Large language models still produce plausible but incorrect answers when uncertain—a long-standing issue affecting model reliability. Training models to abstain is a natural approach, but most methods rely on fixed error penalty coefficients and cannot flexibly adjust behavior at inference time according to different confidence requirements. A recent study proposes RLEC, the first method to enable a single model to respond to explicit confidence targets, opening a more direct path for behavior calibration.

Explicit Confidence Targets: Letting the User Specify "Answer or Not"

Explicit confidence targets mean the user specifies a target value p in the prompt, and the model directly chooses whether to answer or abstain, with the goal that the accuracy of answered questions reaches at least p. Ideally, the same model should understand and respond to different p values: when p is set high, the model answers only the most confident questions; when p is set low, the model expands its answering range.

This setting moves confidence control from external threshold calibration into the model's policy, eliminating the need for additional decision systems or post-hoc threshold adjustments at inference time. The research notes that previous reinforcement learning methods for training abstention mostly used fixed error penalties, and therefore could not adapt to different confidence requirements at inference—exactly the problem explicit confidence targets aim to solve.

Calibration is key background for understanding this article. A model is "well-calibrated" when its claimed confidence matches its actual accuracy: for example, among all responses where it gives 80% confidence, about 80% are correct. Traditional approaches rely on post-hoc threshold calibration—first have the model output confidence scores, then select a threshold on a validation set and abstain below it. The problem is that once the threshold is fixed, retuning is required for different accuracy requirements. Explicit confidence targets internalize this process: the target value p directly enters the prompt and training objective, and the model learns an answering policy that varies with p, thus eliminating the need for external threshold engineering at inference.

Why Previous Attempts Failed

Learning with a single model to respond to different confidence targets is difficult in practice; previous attempts could not even distinguish between different target values. The paper attributes the failure to two issues:

First, the reward signal that distinguishes different p values is often lost during optimization. The model does not receive sufficiently clear gradient information to discern which confidence level it should currently respond to, and different targets tend to homogenize during training.

Second, the raw error penalty grows unboundedly as p approaches 1. That is, when users demand extremely high accuracy, the penalty term inflates to the point of destabilizing training, and the model fails to learn effective behavior.

These two issues—one about whether the learning signal can be preserved, the other about whether the reward scale is controllable—together explain why previous methods failed to distinguish confidence targets.

Reward signal homogenization deserves further explanation. When training abstention with reinforcement learning, the reward typically consists of "points for correct answers, penalties for wrong answers, and an intermediate score for abstention." If the reward scales corresponding to different confidence targets p differ only slightly, or are compressed into similar ranges after advantage normalization, clipping, and similar operations, the model cannot tell from the gradient whether it should be conservative or aggressive—and thus learns nearly identical policies for different p. This is precisely what the paper means by "the reward signal that distinguishes different p values is lost during optimization"—the problem is not insufficient model capacity, but that the distinguishability of the optimization objective is flattened by the training mechanism itself.

Three Key Designs of RLEC

Addressing the above diagnosis, RLEC (Reinforcement Learning with Explicit Confidence Targets) proposes three corresponding designs:

• Preserve the learning signal needed to distinguish confidence targets: Ensure that different p values remain distinguishable to the model during optimization rather than being flattened.
• Introduce a stable reward family: The new family of reward functions avoids the penalty term growing unboundedly as p approaches 1 while preserving the desired answering conditions in expectation.
• Reward scheduling: Strike a balance between effective learning and training stability, so that training both learns something and does not become unstable.

Combined, these three designs make training a single LLM for explicit confidence target control possible for the first time. The paper emphasizes that the core value of these designs is making confidence targets truly part of the trained policy, rather than an external rule at inference time.

Reward scheduling in reinforcement learning training refers to letting the reward function change dynamically over the course of training rather than remaining fixed throughout. A common approach is to use smoother, smaller-scale rewards early on to stabilize exploration, then gradually tighten them to approach the target behavior. In the context of RLEC, scheduling reconciles two conflicting needs: although the reward family avoids unbounded growth as p→1, overly conservative penalties prevent the model from learning the aggressive behavior of "answering when it should"; conversely, overly strong penalties cause training instability. By adjusting reward strength or composition during training, the model can first establish a basic answer/abstain distinction, then gradually align to fine-grained confidence targets.

Experimental Performance: Balancing General Capabilities and Calibration

The study conducted experiments on multiple benchmarks and across different model scales. Results show that RLEC largely preserves the model's general capabilities while achieving competitive calibration performance. For such training methods, "not sacrificing original capabilities" and "effective calibration" are often hard to achieve simultaneously, and RLEC performs well on both.

Another notable feature is fine-grained target control: after training, the model can directly respond to different confidence requirements without an external decision system or post-hoc threshold calibration. This is clearly distinct from traditional methods relying on fixed penalties—which typically require additional adjustment mechanisms to change confidence requirements at inference time.

The paper summarizes the significance of this achievement as follows: incorporating explicit confidence targets into the trained policy provides a more direct new path for behavior calibration. From an application perspective, this means users can directly express at the prompt level "I want answers with this level of accuracy," and the model autonomously decides whether to answer or abstain, without relying on external confidence threshold engineering.

Brief Commentary

RLEC's value lies in not stopping at the vague goal of "training models to learn abstention," but in parameterizing the confidence target itself and diagnosing the specific mechanisms by which previous methods failed—reward signal loss and unbounded penalty growth. Correspondingly, it offers three designs—preserving the signal, stabilizing the reward, and reward scheduling—which are clear in thinking and mutually complementary.

In terms of limitations, the original abstract does not disclose specific benchmark names, model scale ranges, or calibration metric values, and the experimental coverage and comparison baseline details await confirmation from the full text. Additionally, whether explicit confidence targets can remain stable in more complex multi-turn interactions or tool-calling scenarios is currently unclear. But on the point of "a single model responding to fine-grained confidence requirements without an external threshold system," this work provides a direction worth following for behavior calibration of trustworthy LLMs.

One-line Summary

RLEC uses three designs to enable a single model to respond to confidence targets in prompts and autonomously decide whether to answer or abstain.

Paragraph Summary

The article introduces a reinforcement learning method called RLEC, aimed at solving the reliability issue of large language models still giving wrong answers when uncertain. Traditional abstention training relies on fixed error penalty coefficients and cannot adapt to different confidence requirements at inference time. RLEC proposes explicit confidence target setting: the user specifies an accuracy rate p in the prompt, and the model decides whether to answer or abstain accordingly, with the same model able to respond to different p values. The paper diagnoses two reasons previous methods failed—the reward signal distinguishing different p values is lost during optimization, and the error penalty grows unboundedly as p approaches 1—and correspondingly offers three designs: preserving the learning signal, a stable reward family, and reward scheduling. Experiments show that the method achieves competitive calibration performance while preserving general capabilities, and without the need for an external threshold system.