MAS: A New Model-Agnostic Variable Selection Method with FDR Control Without Covariate Distribution Assumptions

In today's increasingly complex machine learning models, how to screen out truly meaningful covariates from massive features while controlling the false discovery rate (FDR) has always been a core challenge in interpretable scientific discovery. Traditional methods often rely on specific model assumptions, and although Model-X knockoffs provides a general framework that leverages covariate distribution information, its dependence on distributional knowledge also limits practical applications. A new study proposes a complementary method called MAS (Model-Agnostic Selection with agent-X importance), attempting to circumvent this limitation.

Why Variable Selection Needs New Approaches

Variable selection plays a key role in statistics and machine learning. Whether searching for disease-causing genes in genomics or identifying key indicators in financial risk control, researchers hope to pick out variables that truly affect outcomes from a large pool of candidates while avoiding introducing too many false positives.

Model-X knockoffs is a method that has attracted much attention in recent years. It achieves FDR control under the premise of a known covariate distribution by constructing "imitation" variables related to the original covariates. However, in reality, the distribution of covariates is often unknown or difficult to estimate accurately, which has become a major obstacle to the deployment of this method.

MAS was proposed precisely to fill this gap. Unlike Model-X knockoffs, MAS does not require knowledge of the covariate distribution. Instead, it achieves model-agnostic variable selection through a sample splitting strategy and a novel variable importance measure.

False Discovery Rate (FDR) is a core concept in multiple hypothesis testing, proposed by Benjamini and Hochberg in 1995. When researchers simultaneously test hundreds or thousands of hypotheses (for example, testing the association between tens of thousands of genes and a disease in genomics), false positives accumulate rapidly if no correction is applied. FDR is defined as the expected proportion of false discoveries among all findings declared significant. Compared with the stricter family-wise error rate (FWER), FDR control is more lenient, allowing a certain proportion of false positives to exist, thereby preserving higher statistical power while ensuring reliability. In variable selection problems, each candidate variable corresponds to a null hypothesis (that the variable has no effect on the outcome). FDR control means that among the finally selected set of important variables, the proportion of truly irrelevant variables is limited to within a preset threshold (such as 5% or 10%). This makes FDR the most commonly used error control criterion in high-dimensional variable selection.

Model-X knockoffs was proposed by Barber and Candès in 2015. It is a variable selection framework that achieves exact FDR control under the condition that the covariate distribution is known. Its core idea is to construct a "imitation" variable (knockoff) for each original covariate, such that the joint distribution of the knockoff variables and the original variables satisfies an exchangeability condition—that is, the joint distribution remains unchanged after swapping the positions of the original variables and their corresponding knockoffs. By comparing the importance statistics of the original variables and the knockoffs in the model, the knockoff+ threshold can be used to precisely control FDR. The advantage of this method is that it does not depend on the model form of the outcome variable Y; it only requires that the distribution of the covariates X is known. However, in practical applications, the distribution of X is often unknown or difficult to estimate accurately (especially under high-dimensional or complex data structures), and distribution estimation error directly undermines the theoretical guarantee of FDR control. This has become the main bottleneck for the deployment of Model-X knockoffs.

Agent-X Importance: The Core Innovation

The core of MAS lies in its definition of "agent-X importance." This measure evaluates, at the population level, the difference in impact on "oracle predictiveness" caused by the presence or absence of a certain variable—that is, it compares the original distribution with the distribution after replacing a certain variable with an "agent variable," and examines the gap in predictive ability between the two.

In the specific implementation, the researchers construct the agent variable as some constant and adopt a cross-fitting strategy to efficiently estimate this importance. This approach cleverly transforms the problem of distributional assumptions into an operational estimation problem, thereby bypassing dependence on the covariate distribution.

Intuitively, if a variable is truly important, then after replacing it with a constant, the model's predictive ability should decline significantly; conversely, if the variable is irrelevant, the predictive ability will not change much after replacement. MAS uses precisely this logic to identify important variables.

Cross-fitting is a sample splitting technique used to eliminate overfitting bias, widely applied in debiased machine learning and doubly robust estimation. Its basic procedure is: randomly divide the data into K folds; for each fold, use the data from the remaining K-1 folds to train the model, then compute predictions or estimators on the held-out fold, and finally aggregate the results across all folds. The key benefit of doing this is to ensure that the model used for estimation and the data used for evaluation are independent of each other, thereby avoiding overfitting bias caused by using the same data for both training and evaluation. In MAS, cross-fitting is used to efficiently estimate agent-X importance: after the model is trained on part of the data, the difference in predictive ability before and after variable replacement is evaluated on another part of the data, ensuring the asymptotic normality of the importance estimator and providing a theoretical basis for subsequent FDR control.

Resampling and Data Aggregation: Improving Statistical Power

An importance measure alone is not enough. To maintain sufficient statistical power while controlling FDR, MAS introduces two key strategies.

The first is the "resplit-per-covariate" strategy. Traditional sample splitting leads to dependence among estimators, affecting the accuracy of FDR control. By independently resampling for each covariate, MAS effectively decouples this dependence.

The second is data aggregation technology. A single split often wastes data and reduces power, but by aggregating the results of multiple splits, MAS can enhance detection capability without sacrificing FDR control. The study points out that the combination of these two strategies makes MAS competitive in both computational efficiency and statistical power.

Sample splitting is a general strategy of dividing data into independent subsets for use in different estimation steps. In variable selection, a typical approach is to divide the data into two parts: one part is used to fit the model or compute importance statistics, and the other part is used to determine the selection threshold. The advantage of sample splitting is that it eliminates overfitting in the selection process, making the theoretical derivation of FDR control more concise. But its cost cannot be ignored: after splitting, the amount of data in each part decreases, leading to reduced estimation accuracy and lower statistical power. In addition, if splitting is performed separately for multiple variables, the estimators for different variables may become dependent due to shared data, interfering with joint FDR control. Through "resplit-per-covariate," MAS independently splits for each variable, retaining the decoupling advantage of sample splitting while avoiding the dependence problem among variable estimators, and then combines data aggregation to compensate for the power loss of a single split.

Theoretical Guarantees and Experimental Validation

Under mild conditions, the researchers established theoretical guarantees for asymptotic FDR control for MAS. This means that when the sample size is sufficiently large, MAS can control the false discovery rate within a range close to the target level.

Extensive simulation experiments and real-data analyses further verified the effectiveness of the method. The results show that MAS performs comparably to or even better than existing methods in three dimensions: FDR control, computational efficiency, and statistical power. One detail in particular: because it does not require covariate distribution assumptions, MAS has a lower barrier to entry and a broader scope of application in practical use.

Implications for Practitioners

For data science practitioners, the value of MAS lies in that it provides a "plug-and-play" variable selection solution. You do not need to know the distribution of covariates in advance, nor do you need to worry about whether model assumptions hold. You only need to construct agent variables and perform cross-fitting to obtain variable selection results with FDR guarantees.

Forming a complement to Model-X knockoffs, MAS gives researchers more choices in different scenarios: when the covariate distribution is known or can be accurately estimated, knockoffs may be the more efficient choice; when the distribution is unknown or difficult to model, MAS demonstrates its model-agnostic advantage. This complementary relationship itself also drives the development of variable selection methodology.

As machine learning models become more deeply applied in scientific discovery, methods like MAS that balance theoretical rigor and practical flexibility are expected to become an important part of researchers' toolkits.

One-line Summary

MAS is a model-agnostic variable selection method that achieves FDR control without covariate distribution assumptions through agent-X importance and sample splitting.

Paragraph Summary

The article introduces a new variable selection method called MAS, aimed at solving the dependence of traditional methods (such as Model-X knockoffs) on covariate distribution assumptions. The core innovation of MAS is the "agent-X importance" measure: by replacing the target variable with a constant, it evaluates changes in predictive ability to judge variable importance, and uses a cross-fitting strategy for estimation. To balance FDR control and statistical power, MAS introduces resplit-per-covariate to decouple estimator dependence and data aggregation to enhance detection capability. The study establishes theoretical guarantees for asymptotic FDR control under mild conditions. Simulations and real-data experiments show that MAS performs comparably to or even better than existing methods in FDR control, computational efficiency, and statistical power. MAS complements Model-X knockoffs and is suitable for practical scenarios where the covariate distribution is unknown or difficult to estimate, providing data science practitioners with a plug-and-play variable selection solution.