Membership Inference Attacks on Tabular Data: An SoK Reexamination of Privacy Evaluation

In the field of machine learning privacy evaluation, Membership Inference Attacks (MIAs) have long dominated. However, a systematization of knowledge (SoK) paper titled "SoK: Rethinking Membership Inference in Tabular Machine Learning" points out that existing evaluations almost exclusively focus on high-dimensional datasets, leaving the privacy behavior of small- and medium-sized tabular data underexplored. By constructing a new MIA taxonomy, the study systematically examines the key factors influencing vulnerability and reveals a troubling conclusion: while current defenses reduce average leakage, they fail to protect the most sensitive individual records.

Why Tabular Data Became a Blind Spot in Privacy Evaluation

Tabular data is widely present in critical domains such as healthcare, finance, and government, and its scale is often far smaller than that of image or text datasets. The authors note that existing MIA research has primarily designed experiments based on high-dimensional datasets, assuming that higher data dimensionality and more complex models lead to greater privacy risk. This assumption has left the privacy behavior of small- and medium-sized tabular datasets systematically unexamined for a long time.

The research team collected multiple tabular datasets of varying sizes and class granularities and deployed four state-of-the-art attack methods across multiple model architectures. They found that the feature distribution of the dataset—especially class granularity and majority class proportion—has a significant impact on MIA success rates. Specifically, datasets with higher class granularity and a larger proportion of majority class samples are more susceptible to membership inference attacks. This means that privacy risk depends not only on the model itself; the statistical properties of the dataset also play a key role.

Membership Inference Attacks (MIA) are a class of attacks that determine whether a specific sample was present in the training set by observing the model's output for that sample (such as prediction confidence, loss values, or gradient information). The basic assumption is that the model has stronger memorization of training samples and therefore exhibits response patterns different from those of unseen samples. The threat of MIA lies in its ability to infer sensitive information about individuals—for example, whether a patient's medical record was used to train a diagnostic model—without directly accessing the training data, thereby constituting a privacy breach. Since Shokri et al. proposed a membership inference attack framework for machine learning models in 2017, this direction has become one of the core benchmarks in privacy evaluation, but existing studies have generally validated it on high-dimensional image or text data, and its applicability to tabular data still lacks systematic examination.

Reexamining Adversarial Knowledge and Attack Effectiveness

The MIA taxonomy proposed in the paper distinguishes between attacks that rely on adversarial knowledge and reference model-free attacks. Experimental results show that advanced MIA methods are highly dependent on adversarial knowledge about the target model, such as model architecture, training hyperparameters, or output confidence distributions. When such knowledge is limited, attack effectiveness drops noticeably.

In contrast, reference model-free attacks achieve statistically comparable results in most scenarios. This finding challenges the intuition that "more complex attacks are necessarily more dangerous," suggesting that privacy evaluation should focus more on the practical feasibility of attacks rather than merely pursuing greater algorithmic complexity.

Adversarial knowledge refers to the prior information an attacker possesses about the target model or data distribution when carrying out MIA, typically including model architecture, hyperparameters, training algorithms, output confidence distributions, and even the training data of some shadow models. Depending on the degree of knowledge, attacks can be categorized as white-box, gray-box, or black-box. The "reference model-free attacks" emphasized in the paper fall into the black-box category: they do not rely on a reference model trained on the same distribution as the target model, but instead use the target model's own output statistics or simple threshold rules for inference. The practical threat of such attacks is often underestimated because they are easier to carry out in real deployment scenarios: attackers often cannot obtain internal model information, but can obtain sufficient feedback through API queries. Therefore, when assessing MIA risk, the feasibility of an attack should be considered as important as its theoretical complexity.

Single-Out Records: The Vulnerable Group Under Defense Systems

The paper's most striking finding concerns records with unique signatures, the so-called "single-outs." These samples are unique in the dataset, and their very existence may be identifiable by attackers. Experiments show that even when common defenses are applied (such as regularization, differential privacy noise, output perturbation, etc.), single-out records remain highly vulnerable.

The authors emphasize that current defense mechanisms mainly optimize average leakage metrics while neglecting protection for the most sensitive individuals. In healthcare or financial scenarios, single-out records often correspond to rare diseases or special transactions, and once their membership is inferred, they may cause severe privacy harm. This conclusion poses a direct challenge to privacy protection practice: improvement in average metrics does not equate to the security of individual privacy.

Single-out records refer to individuals in a dataset that have unique feature signatures and cannot form an effective anonymity set with other samples. In tabular data, such records usually manifest as rare combinations of categorical features or extreme values of continuous features—for example, the full combination of clinical indicators of a patient with a rare disease, or a transaction whose amount and merchant type are both anomalous. Due to the lack of "cover" from similar samples, the model is more likely to develop overfitting-style memorization of them, and an attacker can infer their membership with high confidence simply by judging whether the model output is anomalous. Although defenses such as differential privacy blur the overall distribution by adding noise, they find it difficult to completely eliminate the unique traces left by single-out records in model parameters. Therefore, the privacy risk of single-out records is fundamentally at the individual level and cannot be reflected by average leakage metrics.

Implications for Privacy Evaluation and Defense Design

This SoK study conveys several key signals. Privacy evaluation should not default to conclusions drawn from high-dimensional datasets, but must be recalibrated for the characteristics of tabular data. Data statistics such as class granularity and majority class proportion should be incorporated into risk assessment frameworks. Defense design needs to shift from "reducing average leakage" to "protecting the most vulnerable records"; otherwise, the privacy of single-out individuals will remain exposed.

The authors call on the community to rethink the evaluation paradigm of MIA in tabular machine learning and to establish benchmarks that are closer to real deployment scenarios. For practitioners engaged in privacy protection research or compliance practice, this systematization provides clear directions for improvement: pay attention to data distribution, scrutinize assumptions about attack knowledge, and design targeted protections for single-out records.

One-line Summary

An SoK paper reveals a blind spot in MIA evaluation for tabular data: defenses only reduce average leakage, while single-out records remain highly vulnerable.

Paragraph Summary

A systematization of knowledge paper reexamines membership inference attacks in tabular machine learning. The study points out that existing evaluations overfocus on high-dimensional datasets and neglect the privacy behavior of small- and medium-sized tabular data. By constructing a new MIA taxonomy and testing multiple attack methods, the paper finds that datasets with higher class granularity and a larger majority class proportion are more vulnerable to attack; advanced attacks rely heavily on adversarial knowledge, while reference model-free attacks often achieve statistically comparable results. Most troublingly, single-out records with unique signatures remain highly vulnerable even when common defenses such as regularization and differential privacy are applied, because current defenses optimize only average leakage and neglect protection for the most sensitive individuals. The paper calls for privacy evaluation to be recalibrated for the characteristics of tabular data and to shift toward protecting the most vulnerable records.