Conformal Policy Selection: Adding a Safety Valve to AI Decision Deployment with FDR Control

AI policies are increasingly involved in high-stakes decisions in healthcare, finance, education, and public services. But a policy that "works on average" is not necessarily safe for everyone—existing offline policy evaluation and safe policy learning tools mostly provide population-level guarantees, yet struggle to answer the key question of "who does it help and who does it harm." A new paper proposes Conformal Policy Selection, reframing selective deployment as a multiple testing problem and using false discovery rate (FDR) control to provide a statistical safety valve for individual-level AI decision deployment.

From Population Guarantees to Individual Risk: The Blind Spot in AI Deployment

The current mainstream approach is to train a global policy and then use offline evaluation to estimate its overall benefit. This may hold in an average sense, but the paper points out that some individuals may experience negative treatment effects. In a medical setting, this means a treatment recommendation policy that performs well across the overall population may systematically give harmful recommendations for a particular high-risk subgroup.

The root of the problem lies in the misalignment between evaluation goals and deployment goals: offline evaluation answers "is this policy good on average," while responsible deployment needs to answer "for which units is deployment safe." The paper turns the latter into an actionable statistical problem—for each candidate unit, test whether its outcome under the AI policy exceeds a user-specified threshold, and control FDR over the selected set.

False discovery rate (FDR) is a statistical metric in multiple hypothesis testing that controls the proportion of false rejections, defined as the expected proportion of rejected null hypotheses that are actually true. Unlike the traditional family-wise error rate (FWER), which controls the probability of making at least one false rejection, FDR allows a certain proportion of false discoveries to be tolerated across a large number of tests, thereby significantly increasing testing power while preserving statistical rigor. In the AI policy deployment setting, each candidate unit corresponds to a null hypothesis (e.g., "the treatment effect of this unit under the policy does not exceed the threshold"), and FDR control means that among the units included in the deployment list, the expected proportion of erroneous inclusions (i.e., units that do not actually satisfy the safety condition but are selected) is bounded at a user-specified level (e.g., 5% or 10%). This framework is particularly suited to decision-making shaped as "screening" rather than "confirmation," because the deployment list itself allows a small number of errors, but there must be an explicit statistical constraint on their proportion.

Method Core: Weighted Conformal p-Values + Two-Step Screening

The technical approach of Conformal Policy Selection consists of two components. The first is weighted conformal p-values, used to correct the distribution shift from the behavior policy to the target policy. Conformal inference itself provides finite-sample validity guarantees, and the weighting mechanism makes it applicable to counterfactual evaluation settings where the behavior distribution and target distribution differ.

The second is a two-step selection and pruning procedure: first select candidate units using a relatively lenient criterion, then remove those that do not satisfy the FDR constraint through a pruning step. The paper proves that this procedure controls FDR in finite samples and establishes a doubly robust asymptotic guarantee—FDR control still holds as long as either the outcome model or the propensity model is correctly specified. This property is especially important in practice, because under high-dimensional covariates it is almost impossible for both models to be correct simultaneously, and double robustness provides room for error tolerance.

Conformal inference is a distribution-free statistical inference framework whose core idea is to use "exchangeability" to construct prediction intervals or p-values, thereby providing finite-sample validity guarantees. Specifically, given a calibration set and a new test point, conformal methods generate a p-value by comparing the nonconformity score of the test point with those of the calibration points; under the null hypothesis this p-value follows a uniform distribution (or satisfies super-uniformity), and can therefore be used directly for hypothesis testing. In causal inference or policy evaluation, because observational data come from a behavior policy rather than the target policy, the data distribution is shifted and standard conformal methods no longer apply directly. Weighted conformal p-values reweight the calibration set by introducing importance weights (the ratio of the probability of observing the unit under the behavior policy to the probability under the target policy), restoring exchangeability under the weighted distribution and thus extending conformal inference to counterfactual evaluation settings. This weighting mechanism is the key bridge connecting offline data with target policy deployment.

Doubly robust estimation is a classic technique in causal inference for handling confounding bias. Its core idea is to combine an outcome model (a model that predicts outcomes given covariates and treatment) and a propensity model (a model of the probability of receiving treatment given covariates) to construct an estimator that is consistent as long as one of the two models is correctly specified. In Conformal Policy Selection, double robustness is used to establish the asymptotic guarantee for FDR control: weight estimation depends on the propensity model, while treatment effect estimation depends on the outcome model, and the paper proves that FDR control still holds as long as either one is correct. This property is especially important in high-dimensional covariate settings, because it is almost impossible for both models to be correct simultaneously; double robustness provides tolerance for model misspecification, making the method more reliable in practice.

Experimental Validation: Robustness Under Four Types of Misspecification

In synthetic experiments, the paper covers four model misspecification scenarios and validates the above theoretical guarantees. Even more convincing is a semi-synthetic study based on real electronic health record covariates: a globally trained policy exhibits misgeneralization on a high-risk subgroup, and FDR control acts as a deployment safety gate, limiting the number of harmful treatments the policy would otherwise have administered.

This work echoes several recent directions in AI safety. Research on model uncertainty and calibration focuses on prediction reliability, while Conformal Policy Selection adds a guarantee at the decision level—not only must predictions be trustworthy, but the screening of deployment decisions must also have statistical constraints. Compared with simple confidence-interval filtering, the advantage of the multiple testing framework is that it explicitly controls the proportion of errors in the selected set, which is closer to the actual decision form of a "deployment list."

Limitations and Applicable Boundaries Worth Noting

The effectiveness of the framework depends on several premises: users need to set outcome thresholds, and threshold selection itself involves domain judgment; the weight estimation for conformal p-values depends on the quality of the propensity model, although double robustness mitigates this issue. In addition, FDR control is an average guarantee over the selected set, not a deterministic safety promise for each individual—it constrains the expectation of the "proportion of errors in the selected set," rather than eliminating all individual risk.

For high-stakes AI deployment practice, the implication of this method is that rather than pursuing a single globally optimal policy, it is better to treat deployment as a screening problem and use statistical guarantees to delineate the range of trustworthy interventions. This line of thinking has direct application potential in precision medicine, targeted educational interventions, and differentiated financial services.

Key Takeaways

• Model targeted deployment of AI policies as a multiple testing problem, testing for each candidate unit whether the treatment effect exceeds a threshold.
• Weighted conformal p-values correct the distribution shift from the behavior policy to the target policy, and a two-step screening procedure achieves finite-sample FDR control.
• Doubly robust asymptotic guarantee: FDR control is maintained as long as either the outcome model or the propensity model is correct.
• Semi-synthetic electronic health record experiments show that FDR control can serve as a deployment safety gate, limiting the harm from misgeneralization on high-risk subgroups.

One-line Summary

The paper proposes Conformal Policy Selection, turning targeted deployment of AI policies into a multiple testing problem and achieving finite-sample FDR control through weighted conformal p-values and two-step screening, providing a statistical safety valve for individual-level deployment.

Paragraph Summary

This article introduces a new method called Conformal Policy Selection, aimed at solving the problem that when AI policies are deployed in high-stakes domains such as healthcare and finance, population-level evaluation cannot guarantee individual safety. The method reframes selective deployment as a multiple testing problem, testing for each candidate unit whether its treatment effect exceeds a user-specified threshold, correcting the distribution shift between the behavior policy and the target policy through weighted conformal p-values, and then achieving finite-sample false discovery rate (FDR) control through a two-step selection and pruning procedure. The paper establishes a doubly robust asymptotic guarantee, namely that FDR control is maintained as long as either the outcome model or the propensity model is correct. Synthetic experiments and a semi-synthetic study based on real electronic health records show that the method can effectively limit harmful interventions on high-risk subgroups, providing a statistical safety framework at the decision level for responsible AI deployment.