OLMo 2 Controlled Experiment: The Truth About Training Data Membership Inference

Whether a language model has seen a given document during training directly relates to copyright, privacy, and benchmark contamination. However, most membership inference (MI) evaluations for language models collect member and non-member samples after the fact, so success may be an artifact of distribution shift, and failure yields no definitive conclusion. A recent controlled study on OLMo 2 1B reran the official training stage to provide a clearer answer.

Controlled experiment design: eliminating distribution shift with counterfactual controls

The research team reran the official 50-billion-token mid-training stage of OLMo 2 1B, but with one key modification: selected training data blocks were replaced with documents from 7,814 matched document pairs. These pairs were matched on domain and difficulty, and for each pair a coin flip determined which document was inserted into training and which was held out. The original run released by AI2 from the same random seed, identical in data everywhere else, served as a counterfactual reference.

This design directly addresses the pain point of traditional MI evaluation—members and non-members come from the same distribution, and the only variable is whether a document was trained on. The research code is open-sourced on GitHub (the OLMo-Twin project).

Counterfactual reference is a core concept in causal inference, referring to the outcome that would have appeared under the same conditions if only one treatment variable were changed. In this study, the training data of the original OLMo 2 1B model is almost identical to that of the experimental model, with the only difference being whether the training set contains a particular document from a matched pair. Thus, the original model can be viewed as the counterfactual world "if this document had not been inserted into training," while the experimental model is the factual world "document inserted into training." By comparing the likelihood differences between the two models on the same document, researchers can strip away confounders such as document content, domain, and difficulty, and separately estimate the causal effect of the variable "whether trained on." This design is usually impossible in observational studies and only feasible by rerunning the training pipeline and controlling the random seed.

Core findings: a single exposure changes likelihood, but precise identification is difficult

The experiment yielded several noteworthy conclusions. A single training exposure does raise the average likelihood of an unseen document, but it rarely allows identification of which specific document it was. Using the counterfactual model as a reference, the area under the ROC curve (AUC) of the reference ratio reaches 0.767, but applying a single threshold across domains captures only 3.8% of members at a 1% false positive rate.

This means that even with near-perfect control conditions, document-level membership inference still faces enormous challenges. The study further points out that 13 matched document pairs are enough for a paired test to reach 80% statistical power at the 5% significance level for detecting whether a set was trained on—far more realistic than per-document identification.

Area under the ROC curve (AUC) is a common metric for measuring the overall performance of a binary classifier, ranging from 0.5 (random guessing) to 1.0 (perfect separation). In membership inference, AUC represents the probability that, when a member document and a non-member document are drawn at random, the attacker assigns a higher score to the member document. The AUC of 0.767 reported in this paper shows that, under the counterfactual reference, the model does systematically assign higher likelihood to trained documents than to untrained ones, with discrimination clearly better than random guessing. But AUC is a global average metric, and a high AUC does not mean high recall remains at a low false positive rate. Real audit scenarios often require "identifying as many members as possible at a 1% false positive rate," and here a single threshold applied across domains captures only 3.8% of members, exposing the operational fragility of document-level inference.

The choice of reference model is crucial

The study compared the effects of different reference strategies. A reference model trained with another random seed recovered 92% of the AUC advantage of the counterfactual reference relative to random guessing; a reference-free attack, aggregated across domains, differed from random guessing by no more than 0.07. This shows that the quality of the reference model directly determines the upper bound of the attack, but even so, individual differences at the document level still significantly limit inference precision.

Another key data point: after rerunning the full official post-training pipeline on both models, the average effect retained 89%. This suggests that post-training does not substantially erase the traces left by training exposure.

Statistical power refers to the probability that a test correctly rejects the null hypothesis when the alternative hypothesis is true, usually with 80% as an acceptable lower bound. In this paper, the null hypothesis of the paired test is "there is no systematic difference in likelihood between documents in a set and control documents," and the alternative hypothesis is "this set was trained on." Because the per-document effect size is small and high-variance, a test on a single document is almost impossible to give sufficient power; but pooling the likelihood differences of 13 matched document pairs for a paired test achieves 80% power at the 5% significance level. The "13 pairs" here is a key sample-size conclusion: it means set-level detection does not require massive numbers of documents—as long as the control group is carefully matched, even relatively small data sets can be reliably detected, which has direct operational significance for copyright auditing and benchmark contamination detection.

Implications for copyright and privacy assessment

The core message of this study is that membership inference remains unreliable at the document level, but set-level detection is feasible. For copyright litigation or privacy auditing, requiring per-document proof that "the model has seen this document" is statistically very difficult; detecting that "the model's training data contains a particular set" is far more realistic.

The research team has committed to releasing checkpoints, per-token log probabilities, and evaluation documents, which will provide a rare controlled benchmark for future MI research. For practitioners concerned with training data governance, the value of this work lies not in providing an attack tool but in delineating the boundaries of current technical capability.

Post-training typically includes stages such as supervised fine-tuning and preference alignment, which significantly change the model's behavioral distribution, so a natural question is whether the membership signal left during mid-training is erased by post-training. This study reran the full official post-training pipeline and found that the average effect retained 89%, indicating that the traces left by training exposure on likelihood are quite persistent. Note, however, that "average effect retained" is not the same as "per-document identifiability unchanged": post-training may change the absolute level of likelihood and may also have heterogeneous effects across documents. This result suggests that even after post-training, reference-model-based set-level detection remains effective, but document-level attacks do not become easier or harder because of post-training; their fundamental bottleneck remains that the single-exposure effect is too small.

One-line Summary

A controlled rerun of OLMo 2 training shows that document-level membership inference remains unreliable, and only set-level detection is realistic.

Paragraph Summary

This article presents a controlled membership inference study on OLMo 2 1B. Researchers reran the official mid-training, randomly assigning matched document pairs to members and non-members, and used the original run with the same seed as a counterfactual reference to eliminate distribution shift. Results show that a single training exposure raises document likelihood, with an AUC of 0.767 under the counterfactual reference, but a single cross-domain threshold at a 1% false positive rate captures only 3.8% of members, so document-level inference remains difficult. Thirteen matched document pairs suffice for a paired test to reach 80% statistical power, making set-level detection more feasible. Reference models with different seeds recover 92% of the AUC advantage, and reference-free attacks are close to random; official post-training retains 89% of the average effect. The study delineates the current capability boundaries of membership inference.