The Illusion of Membership Inference Attacks: Do Language Models Really Memorize Training Data?

A Tempting but Dangerous Inference

When a language model predicts a sentence with unusual ease — that is, when the sentence has an extremely low perplexity — researchers often reach an intuitive conclusion: this sentence must have appeared in the model's training data. This type of reasoning forms the core logic of Membership Inference Attacks (MIA), and serves as the technical foundation for a range of hot-button topics including privacy auditing, copyright disputes, and data contamination detection.

Yet nearly every published membership inference test shares a fatal methodological flaw: it must "guess" which sentences were in the training data and which were not. Since the pretraining corpora of most models are closed-source, researchers can only rely on heuristics to construct "members" and "non-members" — and this guesswork itself can introduce systematic bias.

A newly published arXiv paper (arXiv:2609.10830) sets out to "remove this guesswork." The authors leverage two model families with publicly available pretraining corpora — OLMo-2 and Pythia — along with a public index covering those corpora that can return the exact number of times any given sentence appears in the data. This verified repetition count gives a direct answer to what was previously a murky question.

A Two-Sided Pincer: Membership Inference Signals Are Nearly Absent

The paper's central finding is described by the authors as a "pincer" — closing in from both sides to squeeze out any remaining space for the claim that "model memorization is detectable."

Ordinary Text: Training Exposure Leaves Almost No Trace

At the repetition levels typical of ordinary text, five models ranging from 1B to 13B parameters carry at most a faint trace of having seen their own training content. To rule out interference from a sentence's inherent fluency and quality, the authors employ a clever approach: reading the same sentence with two different models, using the contrast to cancel out fluency and text quality by construction.

The results are sobering — the rank correlation of this trace is approximately -0.08. On this scale, -1 represents perfect correlation and 0 represents no relationship at all. In other words, across the realistic distribution of text repetition, whether a model has "seen" a sentence and how easily it predicts that sentence have almost no reliable connection.

Highly Repeated Text: Training Exposure Is Indistinguishable from "Fame"

So where do membership inference signals get stronger? The answer: when a sentence's repetition count exceeds roughly one thousand, traces do become significant. But here the second arm of the pincer closes in — both corpora agree highly on which sentences are highly repeated. The reason is simple: these high-repetition sentences tend to be "famous" ones, repeatedly cited and widely circulated across the internet.

This raises an inescapable question: when a sentence appears repeatedly in training data and is widely known in the broader cultural context, does the model's low perplexity come from "training exposure" or from "fame"? The two are no longer distinguishable. A detector may appear to capture a membership signal, but what it's actually capturing is the text's prominence.

How Spurious Membership Signals Get "Manufactured"

The paper further reveals how two common practices artificially manufacture false membership inference signals — a sharp methodological critique aimed at the entire MIA research field.

Word Substitution: Rewarding Author Word Choice, Not Model Memory

One common approach to constructing non-members is to take a member sentence and change one word, producing a near-match that isn't in the training set. Experiments do show that models prefer the original sentence. But here's the key point — this preference gap is the same regardless of whether the original sentence appeared once or a hundred times in the corpus.

This means the model is not rewarding "memory" but rather the author's word choices: the original words fit the natural distribution of language better, while the substituted word makes the sentence feel awkward. This preference has nothing to do with whether the model has genuinely "seen" the sentence before. Only when repetition counts exceed one thousand does the gap begin to grow with model scale — precisely the regime where the pincer closes and training exposure becomes entangled with fame.

The Register Trap in Control Sentences: Inflating AUC from 0.83 to 0.94

Even more telling is the second experiment. When researchers replaced control sentences with ones drawn from a different register than the member sentences, an MIA detector's AUC jumped from 0.83 all the way to 0.94. On this scale, 0.5 is equivalent to random guessing and 1.0 is perfect separation.

An AUC of 0.94 looks like a strong detector — seemingly enough to convince observers that "model memorization is detectable." But the truth is: the detector was never detecting memorization. It was simply detecting stylistic and register differences between member and non-member sentences. Any mismatch in the distribution of control sentences systematically inflates detection performance. This also explains why many past MIA studies reported optimistic detection results — their "success" was likely built on biased control sets.

Implications for Privacy Auditing and AI Copyright Disputes

The significance of this work extends well beyond a technical paper. In recent years — from news organizations suing AI companies over using their content for training, to users worried about personal data being memorized by language models — membership inference attacks have been widely regarded as a technical means of "proving that a model has seen a piece of text."

But this study's conclusions are sobering: at realistic text repetition levels, language models carry barely detectable traces of their own training exposure (rank correlation approximately -0.08). Detection results that appear to work either fall in the high-repetition regime where training exposure and fame are indistinguishable, or are artificially amplified by biased control sets.

This serves as a serious warning that MIA results must be treated with extreme caution when used as legal or policy evidence — a high AUC number may be nothing more than an artifact of stylistic differences, not genuine evidence of model memorization.

To their credit, the authors have released their sentence corpus, repetition counts, and all code, providing a reproducible benchmark for future research. This experimental paradigm — replacing guesswork with verified ground-truth labels — may well be the right direction for MIA research to become genuinely rigorous.

Conclusion

The paper's title is its conclusion: "Detectable Only Where It Is Confounded." Membership signals in language models don't not exist — they only appear strong when entangled with confounding factors such as fame, register, and word choice. Once these confounders are stripped away, the signal genuinely attributable to "memorization" nearly vanishes. For anyone attempting to use perplexity as evidence that "a model has seen a piece of text," this is a reality that cannot be ignored.