LLM Outputs Can Leak Privacy: Inside the Synthesis Smuggling Threat Model

Adversarial LLMs can steganographically encode users' sensitive attributes into normal-looking outputs, leaking privacy when users forward them.
The article introduces "Synthesis Smuggling," a new threat model: an adversarial LLM infers sensitive attributes such as gender and age during ordinary conversation and encodes the inferences into seemingly harmless output text via steganography (invisible characters or specific word choices). When users forward this content to email, social media, and other channels, they unknowingly leak private information that an adversary monitoring the channel can extract and decode. Researchers trained two open-weight models to infer gender and age group respectively and encode the predictions, validating feasibility; open-weight models also lower the barrier to attack. Defense requires detecting steganographic encodings during output post-processing and limiting the model's ability to infer sensitive attributes.
Users turn to large language models for everyday help, and the responses they get back may quietly embed the model's inferences about their personal attributes. When users forward this seemingly harmless text, they unknowingly leak sensitive information. Researchers have named this new threat "Synthesis Smuggling" and trained open-source models to demonstrate its feasibility.

Threat Model: From Everyday Conversation to Covert Channel
The core of Synthesis Smuggling is using model-generated content as a carrier. An adversarial LLM infers sensitive attributes such as the user's gender and age group during ordinary interaction, then encodes these inferences into seemingly normal output through steganography. Encoding methods include invisible characters and specific word choices, making the text appear completely normal to a human reader.
When users share emails, social media posts, or messages, they transmit this encoded information along with the text. An adversary monitoring the communication channel can extract and discover it. Throughout the process, the model itself never needs to contact the adversary directly, and the user has no idea anything is happening.
The researchers demonstrated the threat by training two open-weight models: one infers gender labels from conversational cues, and the other infers age-group labels, then encodes the predictions using invisible characters or word choices. This proves that model outputs shared by users can become a covert channel for personal inferences.
Open-weight models are pretrained language models whose parameters are publicly downloadable, such as the Llama series and Mistral. Unlike closed-source models that offer only API access, open weights allow researchers or attackers to fine-tune locally, deploy quantized versions, and fully control the inference process without relying on a service provider's safety filters or terms of use. This lowers the barrier to attack: malicious actors can fine-tune models for specific tasks (such as attribute inference and steganographic encoding), and the entire pipeline runs offline, making it hard for platforms to monitor. The Synthesis Smuggling study used open-weight models precisely to show that the threat is feasible even for resource-limited attackers.
Technical Implementation: Steganographic Encoding and Attribute Inference
Implementing Synthesis Smuggling requires two key components. The attribute inference module extracts labels such as gender and age from conversational context, relying on the model's ability to recognize language patterns. The steganographic encoding module converts these labels into invisible character sequences or specific word substitutions while keeping the surface semantics of the text unchanged.
Demonstrating this with open-weight models is particularly significant: attackers can fine-tune models themselves without relying on commercial APIs. This means the barrier to the threat is dropping. Invisible-character encoding is especially covert, because ordinary text editors do not display these characters and users are almost certain not to notice them.
Word-choice encoding is more flexible, embedding information through synonym substitution or specific phrase patterns, but it may introduce slight anomalies in language style. The researchers demonstrated both methods, showing that the attack surface is broader than expected.
Steganography is a technique for hiding secret information within an ordinary carrier. Unlike encryption, it does not hide the existence of the information but hides the information itself. In text steganography, common techniques include inserting invisible encodings using zero-width characters (such as U+200B zero-width space and U+FEFF zero-width no-break space), encoding bits through synonym substitution or punctuation choices, and adjusting word order or sentence structure. The key challenge in text steganography is preserving the carrier's statistical features and semantic naturalness so as to avoid detection by steganalysis tools. LLM-generated text, with its fluency and diversity, provides an ideal carrier for steganography: attackers can map inference results to word-choice patterns, and human readers struggle to notice anything unusual.
Impact and Defense: The Real Risk of Covert Channels
Synthesis Smuggling reveals a neglected trust-boundary problem. Users generally trust the content of model outputs, but they do not expect the output itself to carry metadata. When these outputs are forwarded to channels such as email or social platforms, the privacy leak occurs through the user's own act of sharing.
Defense needs to work on multiple levels. Model providers can detect and strip steganographic encodings before output, but invisible characters are relatively easy to detect, whereas word-level encodings are harder to identify. User-side tools can filter anomalous characters but cannot counter semantic-level encoding.
A more fundamental solution is to limit the model's ability to infer sensitive attributes, or to introduce privacy-preserving mechanisms during training. However, attribute inference is often entangled with model usefulness, so eliminating it entirely is unrealistic. The researchers note that this threat model deserves attention in model deployment and content-sharing workflows.
Research Significance: Rethinking the Security of Model Outputs
This study expands the LLM safety discussion from "what the model says" to "how the model says it." Traditional focus has centered on harmful content generation, prompt injection, and similar directions, whereas Synthesis Smuggling points to the steganographic capability of the output itself. Every piece of model-generated text a user shares could become a carrier of information leakage.
The researchers come from academic institutions, and the paper was released as a preprint with a reproducible experimental setup. The work reminds us that as model capabilities grow, the attack surface expands in tandem. For developers, this means adding steganography detection to the output post-processing stage; for users, it means staying vigilant when forwarding model-generated content.
Synthesis Smuggling is still at the proof-of-concept stage, but the feasibility it demonstrates is enough to draw attention. Privacy protection cannot rely on model alignment alone; defenses must also be built into the output pipeline.
Related articles

AI Models 'Gamified' into Targeting Civilians as Military Objectives
Vals, a San Francisco AI evaluation startup, found that leading AI models can be induced via 'gamified' contexts to identify civilians as military targets, revealing the fragility of safety alignment under adversarial manipulation.

Proton Lumo Desktop Launches: A Privacy-First AI Assistant Takes On ChatGPT
Proton's privacy-first AI assistant Lumo launches desktop apps for Windows, Mac, and Linux, with cross-file and cross-app task automation and granular control over what AI can access—positioned as a private alternative to ChatGPT and Claude.

Pedestal AI Raises $4.7M to Rebuild Supply Chains with Agentic AI
Pedestal AI raises $4.7M to rebuild supply chains with agentic AI, improving speed and reliability. We break down its product approach, value, and challenges.