Calibration and Risk-Budget Decisions for System One Models: A New Framework for Binary Judgments in LLM Agents

LLM workflows and agents rely on a large number of binary decisions in their daily operations: allow or block an input, keep or discard a retrieved document. When these systems have no "hand off to human" option, they must still produce an action even in the face of uncertainty — and traditional probabilistic interfaces expose a fundamental shortcoming here.

Probability scores are not deployment decisions

Most LLM applications, when they need to make a binary judgment, habitually take the probability score output by the model and pair it with a fixed threshold (such as 0.5 or 0.9) to draw the action boundary. The problem with this approach is that the "probability" output by the model is often just a calibrated apparent confidence. It reflects the statistical regularities under the training distribution, not the actual error rate on the specific input at hand.

The paper's core argument is precisely this: relying on a probabilistic interface to drive deployment decisions is insufficient. In scenarios lacking a human fallback mechanism, the system must still take action when uncertain, and what truly needs to be controlled at that point is not classification accuracy or AUC, but the false omission risk on the subset of inputs where a utility-maximizing action has already been taken.

The so-called false omission refers to cases where the system should have taken a certain action but did not — for example, a harmful input that should have been blocked is let through. This risk metric is closer to deployment reality than the overall error rate, because the cost of the portion of inputs handled by the conservative fallback strategy has already been controlled separately.

Method design: conservative fallback plus risk budget

The framework proposed in the study contains two key mechanisms.

Conservative fallback: When the model is uncertain about a judgment, the system by default executes a safe but utility-conservative action. In this way, high-risk decisions are confined to a controllable subset.

Risk-budget selection: On the subset of inputs assigned to the utility-maximizing action, a constraint is imposed on the false omission risk. This transforms the problem from "how high a threshold should be chosen" to "given a risk budget, how many inputs can be allowed to receive an effective action."

The authors validated this on six datasets, covering three types of tasks: content moderation, safety filtering, and document reranking. The comparison targets included a general-purpose System One model, task-specific baselines, and a fine-tuned System One model with locally available open-source weights.

The key experimental conclusion emerged around the threshold selection method: in settings where test risk is definable, the naive threshold selection baseline exceeds the target risk level in roughly half of cases, while the Learn Then Test (LTT) method exceeds it in only 7.8% of cases. LTT's advantage lies in the fact that it leverages finite-sample statistical guarantees rather than relying on point-estimate probability calibration.

False omission risk is a type of conditional risk metric that is contrasted with false discovery risk. In binary decisions, a false discovery means the system took an action but the action was wrong; a false omission means the system should have taken an action but chose the conservative fallback instead.

This concept is closer to deployment reality than the overall error rate because, in allow/block pipelines, the costs of the two types of errors are often severely asymmetric: a mistaken block may only be a loss of user experience, whereas a mistaken allow may cause a safety incident. Measuring false omission risk only on the subset of inputs that have already received a utility-maximizing action is equivalent to acknowledging that the portion of inputs handled by conservative fallback has already been separately backstopped and does not need to be mixed into the same metric. This division allows engineering teams to use a scalar risk budget — for example, "false omission rate no higher than 5%" — as the deployment gate, rather than making vague trade-offs among multiple entangled error rates.

Patterns revealed by systematic comparison

The value of this set of experiments is not only in reporting LTT's low exceedance rate, but also in quantifying the distance between "well calibrated" and "risk controllable." A model may perform well on calibration metrics, yet when it is placed into a real allow/block pipeline, it may still cause actual risk to exceed expectations because of statistical flaws in threshold selection.

The cross-domain comparison across six datasets also shows that this problem is not an accidental phenomenon of some particular task. Although content moderation, safety filtering, and reranking differ in business form, they all involve the same type of decision dilemma: "an action must still be executed under uncertainty." The performance differences among the general-purpose System One model, the task baselines, and the open-source fine-tuned model further point to one conclusion: improving model capability cannot replace risk-control design at the decision layer.

You may not have noticed that risk-budget selection is essentially a trade-off: it compares "coverage of useful actions" and "observed errors" within the same framework. This gives engineering practice an operable adjustment knob — define the error budget you can tolerate, and then let the system process as many inputs as possible under that constraint.

Learn Then Test (LTT) is a threshold selection framework based on multiple hypothesis testing proposed by Angelopoulos et al. Its core idea is to treat all candidate thresholds as a set of hypotheses to be tested, use p-value construction methods under finite samples to compute the risk upper bound corresponding to each threshold, and then use an FDR (False Discovery Rate) control procedure to select in one pass the set of thresholds that satisfies the risk budget.

Unlike traditional point-estimate calibration, LTT does not rely on the premise that "probability scores happen to be accurate." Instead, it provides a risk guarantee that holds under finite samples: as long as the candidate threshold set is fixed in advance, the probability that the threshold selected by LTT exceeds the target risk on the true distribution is strictly controlled. This is especially critical in deployment scenarios — the probabilities output by the model may be well calibrated, but once a threshold is selected by a fixed rule (such as 0.5), it loses statistical guarantees, and the actual risk may spiral out of control due to distribution shift. LTT turns "choosing a threshold" from engineering parameter tuning into a decision problem that can be answered statistically.

Implications for LLM engineering practice

For teams building LLM workflows and agents, this study offers several directly usable judgments.

First, do not directly use the probabilities output by the model as decision confidence. Probability calibration and deployment risk are two independent evaluation dimensions; good performance on the former does not mean the latter can meet requirements.

Second, in scenarios where human handoff is impossible, a conservative fallback strategy should be explicitly designed, and the routing of uncertain inputs should be managed separately rather than mixed with certain inputs on the same threshold line.

Third, threshold selection needs statistical guarantees. The value of finite-sample methods like LTT is that they turn "will the threshold I choose exceed the risk budget" into a question that can be answered statistically, rather than an experiential gamble in offline parameter tuning.

From a broader perspective, this work reframes LLM binary decisions from a "classification problem" into a "resource allocation problem under a risk budget." This shift in perspective may be more important than any single methodological improvement.

One-line Summary

The paper points out that driving deployment decisions with LLM probability scores plus fixed thresholds is unreliable, and proposes a conservative fallback plus risk-budget framework implemented with LTT to make risk controllable.

Paragraph Summary

This study focuses on the binary decision problem of LLM workflows and agents when there is no human handoff option. It points out that the existing practice of directly using model-output probability scores with a fixed threshold to choose actions leads to actual risk exceeding the target level in about half of the test scenarios. The paper proposes two core mechanisms: a conservative fallback strategy that makes uncertain inputs default to a safe but conservative action, and risk-budget selection that constrains false omission risk on the subset of utility-maximizing actions. After validation on six datasets across three types of tasks — content moderation, safety filtering, and document reranking — the Learn Then Test (LTT) method exceeded the target risk in only 7.8% of settings, significantly outperforming naive threshold selection. The study further points out that improving model capability cannot replace risk-control design at the decision layer, and that LLM binary decisions should be reframed from a classification problem into a resource allocation problem under a risk budget.