Bayesian Low-Rank Posterior: Single-Training Membership Inference for Large-Scale Vision-Language Models

Replaces shadow model ensembles with a LoRA-subspace Bayesian posterior, enabling scalable VLM membership inference from a single fine-tuning run.
This paper tackles the prohibitive cost of training reference models for membership inference on large vision-language models by modeling Bayesian uncertainty within the LoRA subspace. Using SWAG, it builds a posterior from one fine-tuning run and samples virtual reference models to estimate score distributions cheaply. Evaluated on three VLMs for document and medical VQA, it outperforms standard reference attacks while requiring only a single run; limitations include reliance on the LoRA-subspace assumption.
Why Membership Inference Needs a New Approach
Membership inference attacks (MIA) attempt to determine whether a given sample was part of a target model's training data, making them a core tool for assessing privacy risk. Current state-of-the-art MIAs rely on reference models (or shadow models) to estimate the score distribution under the target condition, but training such model ensembles is computationally prohibitive for modern large-scale vision-language models (VLMs). According to the original paper, this bottleneck severely limits the scalability of MIA in real-world settings.
The work presented here proposes a scalable alternative: replacing explicit reference model training with Bayesian approximation. The core idea is straightforward—model uncertainty only within the LoRA (Low-Rank Adaptation) subspace, construct a posterior from a single training run, and sample a diverse set of "virtual reference models" from it.

Reference Models and Shadow Models: In membership inference attacks, the attacker typically trains a set of models with architectures and training distributions similar to the target model, called reference models or shadow models. By observing the output score distributions of known members and non-members on the reference models, the attacker can fit a decision boundary and then transfer it to the target model to determine whether a sample was used in training. The effectiveness of this approach depends heavily on the similarity between reference and target models, and requires training multiple models to cover training randomness, so computational cost rises sharply with model size. For vision-language models with enormous parameter counts, the cost of training a full ensemble of reference models often far exceeds the value of the attack itself—this is precisely the fundamental reason a new approach is needed.
Method Core: Doing Bayesian Inference in the LoRA Subspace
The research team strictly limits the scope of uncertainty modeling to the low-rank adaptation parameters rather than the full model weights. The benefit is that posterior construction becomes lightweight and naturally aligns with the practice of parameter-efficient fine-tuning. In their implementation, they use Stochastic Weight Averaging Gaussian (SWAG) to approximate the posterior distribution, thereby efficiently estimating the score distribution under the target condition at a fraction of the cost of traditional shadow model ensembles.
The cleverness of this design lies in the fact that the virtual reference models are not trained independently but sampled from the same posterior. A single fine-tuning run can support the entire attack pipeline, avoiding the compute overhead of training multiple shadow models.
LoRA (Low-Rank Adaptation): LoRA is a parameter-efficient fine-tuning technique that freezes most of the pre-trained model's weights and injects low-rank decomposition matrix pairs (A and B) alongside specific layers, training only these small matrices to adapt to downstream tasks. Since the parameter count of low-rank matrices is far smaller than that of the original model, LoRA significantly reduces the computation and storage overhead of fine-tuning while preserving pre-trained knowledge. Restricting uncertainty modeling to the LoRA subspace means the posterior dimension is on the order of the adaptation parameters rather than the full model weights, which makes Bayesian approximation computationally feasible.
SWAG (Stochastic Weight Averaging Gaussian): SWAG is a posterior approximation method that collects weights from several iterations during stochastic gradient descent, computes their mean and covariance (typically in diagonal or low-rank-plus-diagonal form), and thereby approximates the posterior with a Gaussian distribution. Compared to approaches requiring additional model training, SWAG constructs the posterior using only weight snapshots from a single training trajectory, at extremely low cost. Weights sampled from this Gaussian posterior correspond to different "virtual reference models," providing diverse score distribution estimates for membership inference.
Experimental Design: Three VLMs and Two Privacy-Sensitive Tasks
The paper evaluates the attack on three VLMs adapted to downstream tasks, covering two privacy-sensitive visual question answering tasks: document VQA and medical VQA. These two scenarios have high privacy requirements, as documents and medical images often contain sensitive information, making membership inference more practically meaningful.
To isolate the influence of pre-trained knowledge, the research team also introduced a controlled synthetic dataset. This setup helps determine whether the attack's effectiveness stems from memorization of training members or from knowledge already present during pre-training.
Vision-Language Models (VLMs) and Visual Question Answering (VQA): Vision-language models are multimodal large models that process both image and text inputs, typically consisting of a vision encoder and a language model, capable of answering natural language questions based on image content. Visual question answering is a typical downstream task for VLMs, requiring the model to understand images and generate or select answers. In document VQA, the model must read scanned documents or tables and answer related questions; in medical VQA, the model must interpret medical images and answer clinical questions. The personal identities, medical records, and financial information involved in these two task types give membership inference direct value for privacy auditing.
Synthetic Dataset Control in Membership Inference: Since VLMs may have already seen public data similar to downstream tasks during pre-training, the "membership signal" observed by the attacker may come from pre-training memorization rather than downstream fine-tuning. Introducing a controlled synthetic dataset—i.e., generated data that the model could not possibly have encountered during pre-training—can rule out this confounding factor. If the attack still performs well on synthetic data, it indicates that the method captures membership information introduced during fine-tuning rather than a byproduct of pre-trained knowledge.
Results and Significance: Single Fine-Tuning Beats Standard Reference Attacks
According to the paper's conclusions, the method outperforms standard reference-model-based attacks while requiring only one reference model fine-tuning run. This means membership inference becomes more practical for large VLMs—privacy auditing no longer necessarily comes with expensive ensemble training costs.
From a broader perspective, this work combines Bayesian uncertainty estimation with parameter-efficient fine-tuning, providing a practical technical path for privacy risk assessment of large-scale models.
Limitations and Open Questions
It should be noted that the method's effectiveness rests on the assumption that the LoRA subspace is sufficient to capture key uncertainties. Whether this assumption holds for full-parameter fine-tuning or models under different adaptation strategies remains to be verified. Additionally, experiments focus on visual question answering tasks, and the method's generalization to other downstream tasks is unclear. The original paper also does not discuss the attack's robustness under adversarial defenses.
Key Takeaways
- Replaces explicit shadow model training with a Bayesian posterior over the LoRA subspace, enabling virtual reference model sampling from a single fine-tuning run
- Implements posterior approximation via SWAG, significantly reducing the computational cost of membership inference
- Validated on two privacy-sensitive tasks, document VQA and medical VQA, covering three downstream-adapted VLMs
- Introduces a synthetic dataset to isolate the influence of pre-trained knowledge
- Outperforms standard reference attacks, but applicability to full-parameter fine-tuning scenarios remains to be verified
One-line Summary
Replaces shadow model ensembles with a Bayesian posterior over the LoRA subspace, enabling scalable VLM membership inference from a single fine-tuning run.
Paragraph Summary
This paper addresses the prohibitive cost of training reference models for membership inference attacks on large-scale vision-language models by proposing Bayesian uncertainty modeling within the LoRA low-rank adaptation subspace. The method uses SWAG to construct a posterior distribution from a single fine-tuning run and samples virtual reference models from it, estimating the score distribution under the target condition at a fraction of the cost of traditional shadow model ensembles. The study evaluates on three downstream-adapted VLMs for document VQA and medical VQA, two privacy-sensitive tasks, and introduces a synthetic dataset to isolate the influence of pre-trained knowledge. Results show the method outperforms standard reference attacks while requiring only one fine-tuning run, making membership inference more practical for privacy auditing of large VLMs. The paper also notes that the method relies on the assumption that the LoRA subspace sufficiently captures key uncertainties, and its generalization to full-parameter fine-tuning and other tasks remains to be verified.
Related articles

New AI Governance Initiative: Mandatory Independent Audits and Pre-Deployment Safety Proof
Macron convened government leaders, AI company executives, and researchers to discuss advanced AI risks. Statements call for a rigorous governance framework requiring mandatory independent audits and pre-deployment safety proof.

Roblox Funds Controversial Anorexia Treatment
Roblox funds flow to a controversial creator who promotes ketogenic diets over conventional treatment for anorexia and bulimia, raising medical and regulatory concerns.

AI Safety by Design: European Scientists and Policymakers in Dialogue
European scientists and policymakers discuss the urgency of safe-by-design AI in Berlin and Zurich. This article explains the concept, practical challenges, and implications for global AI governance.