LLM Knowledge Leakage: When AI Pretends to Be a User, Can It Really "Not Know"?

Knowledge Leakage: A Hidden Bias in User Simulation

Using large language models to simulate users in place of human respondents has become a common practice in product research, social simulation, and even recommender system evaluation. But a key assumption is often overlooked: when a model is asked to play someone who "knows nothing about this," can it really hide what it knows?

A recent study formally defines this problem as knowledge leakage—the response bias that arises when knowledge a real user could not possibly possess seeps into simulated answers. The researchers designed a quantitative method and found that even when the context explicitly states "this user has never heard of this item," the model still exhibits a systematic, directionally consistent scoring difference between real items and fictional twin items.

Measurement Method: Real Items and Fictional Twins

The core instrument of this study is elegant: each real item is paired with a surface-form-matched fictional twin item, and a screening process is used to minimize any prior knowledge the fictional item itself might evoke. The score difference between the two thus becomes a diagnostic contrast that can reveal bias even in scoring tasks where there is no "correct answer."

The study set up three knowledge-handling conditions (i.e., low-knowledge prompts of varying strength and explicit declarations of ignorance) and five API endpoints from two model families. The criterion is straightforward: a respondent set up to "not know something" should not give answers that show familiarity with, or awareness of the existence of, that thing.

Fictional twin item is an experimental control design: for each real item (a real product, place name, event, or company), the researchers construct a fake version with a highly similar surface form—for example, replacing a real brand name with a plausible-sounding but nonexistent name, or slightly altering a real place name. Ideally, the model has no prior knowledge of the fictional item, so when scoring it can rely only on the persona information given in the prompt. In this way, any score difference between the real item and the fictional twin can only come from the model's internal knowledge of the real item, not from the persona itself. The purpose of the screening process is to remove samples in which the model might have a prior response to the fictional item due to morphological associations, common naming patterns, or incidental coverage in the training data.

Confidence interval is used in this kind of research to express the uncertainty of an estimate. The reported +0.077 with an interval of [-0.057, +0.204] means that the cross-domain pooled effect is statistically indistinguishable from zero, so one cannot conclude from this that "there is no bias." But after splitting by domain, the resulting -0.359 and +0.438 are each far from zero, indicating that the true effects may be canceled out by pooling and averaging. This is precisely a Simpson's-paradox-like situation: the overall trend and the group trends point in inconsistent directions, reminding readers that they cannot draw conclusions from the pooled number alone.

Two Main Findings

Leakage on Favourable Items Is Hard to Suppress

The first finding is that the tested personas cannot fully suppress signals of the existence and familiarity of real items. Even when the context explicitly states "this persona has never heard of this item," the real-versus-twin scoring contrast for favourable items still persists. In other words, the model verbally agrees that "I don't know," but its scoring behavior reveals that it actually does know.

Bias on Negative Items Reverses by Domain

The second finding is more subtle. When negative-content items are pooled across domains, the contrast value is close to zero (+0.077, confidence interval [-0.057, +0.204]), which looks like there is no bias. But under the same conditions, when the same estimator is split by domain, it yields opposite-direction results: when the rating scale asks whether something has "documented harm," the contrast is -0.359; when it asks about a company or organization, the contrast is +0.438.

This means that pooling negative items together for aggregate analysis masks the opposing differences between domain groups—two effects in opposite directions cancel each other out, creating the illusion that "there is no problem."

Persona conditioning refers to setting a specific identity, background, and knowledge boundary for the model in the prompt so that it answers as that identity. This approach is widely used in user simulation, but its effectiveness depends on whether the model can truly "forget" or "isolate" its own pretrained knowledge. What knowledge leakage research reveals is precisely the incompleteness of this isolation: the model can repeat "I don't know" at the linguistic level, yet leak familiarity signals through indirect behavior such as scoring. In other words, there is a gap between declarative compliance and behavioral consistency. For practitioners, this means persona prompts can only constrain the model's surface-level statements, not reliably constrain its use of internal knowledge; external controls or post-processing are needed to detect and correct this.

What It Means for User Simulation Practice

For teams that use persona-conditioned responses to conduct respondent-level simulation, this study raises a methodological warning. Model knowledge leakage is not random noise; it is a directional, structured bias, and its sign can flip depending on the domain and wording of the scoring question. Simply writing "assume you don't know this product" in the prompt is not enough to solve the problem.

The researchers also candidly state the boundaries of their conclusions: all estimates rest on the adequacy of the twin-item construction and screening, and interpretation at the population level still requires a human low-knowledge baseline. This also points to future work—quantifying knowledge leakage requires human-machine comparison, not the model's self-declaration.

One-line Summary

This study quantifies knowledge leakage bias in LLM user simulation and finds that ignorance declarations in prompts are insufficient to suppress the model's prior knowledge.

Paragraph Summary

This study defines "knowledge leakage" as the response bias produced when knowledge a real user could not possibly possess seeps into simulated answers, and designs a quantitative method centered on comparing real items with fictional twin items. The experiment sets up three knowledge-handling conditions and five API endpoints from two model families, and finds that even when the persona explicitly states it has never heard of an item, the real-versus-twin scoring difference for favourable items still persists. More subtly, the bias for negative items is close to zero when pooled across domains, but takes opposite signs when split by domain, so aggregate analysis masks opposing differences. The study concludes that model knowledge leakage is a directional, structured bias, that simply writing "assume you don't know" in a prompt is insufficient, and that user simulation practice must be alert to this structural bias; it also calls for human-machine comparison against a low-knowledge baseline.