Decoding Subliminal Prompting: Causal Depth and Multi-Token Confounds

How Language Models Covertly Transmit Hidden Features

Subliminal learning reveals a troubling phenomenon: language models can covertly transmit hidden features through outputs that appear unrelated to the target concept. A classic example is a model that, when discussing numbers, implicitly associates them with specific animal concepts. This ability to "smuggle" information poses serious challenges for model safety and interpretability.

One mainstream explanation proposed by researchers is token entanglement — the idea that animal tokens and number tokens share some form of association in the model's output vocabulary, creating a covert transmission channel. A new study published on arXiv (arXiv:2609.19149v1) systematically deconstructs this hypothesis, attempting to pinpoint exactly which mechanism is at work.

The concept of Subliminal Learning originates from research into the implicit encoding of information in language model output distributions. Specifically, when a model is asked to generate outputs related to one concept (e.g., numbers), the token sequences it produces may systematically lean toward another category (e.g., specific animals) at the statistical level — a bias that is not obvious to a casual observer. Researchers typically quantify this hidden association by designing probe classifiers or scoring functions: if the semantic distribution of words chosen while generating number-related content consistently tilts toward some animal concept, the model is considered to be "smuggling" that feature.

The Token Entanglement hypothesis argues that this phenomenon is rooted in the geometric structure of the model's output layer (the Unembedding Matrix) — tokens from different semantic categories lie close together in high-dimensional vector space, causing activations targeting one category to "leak" into neighboring ones. While intuitively plausible, the central challenge posed by this paper is: can the static geometry of the output layer truly explain dynamic behavior at runtime?

Four Distinct "Measurement Questions" Being Conflated

The researchers point out that prior measurements actually answer four different questions, yet are routinely treated as one and the same:

• Whether outputs co-vary
• Whether fixed output vectors are aligned
• Whether an answer can be read out from hidden states
• Whether that hidden state causally controls the answer

These four dimensions correspond to "fixed geometry," "observational readability," "causal timing," and "multi-token measurement" — each an independent property of this frozen prompting channel. Conflating them is precisely why earlier research struggled to localize the true mechanism.

The study adopts a fixed animal-number prompting protocol and measures each dimension separately on the Llama-3.1 model family (from 8B to 70B), aiming to fully decouple these properties.

The Explanatory Power of Fixed Geometry Is Declining

The first experimental finding is striking: as models scale from 8B to 70B, the predictive power of fixed output vector similarity for behavior actually worsens. The paired mean correlation change is -0.080 (95% CI [-0.127, -0.035]). That said, fixed output-head readouts show no clear change in normalized depth AUC.

This means that relying solely on the static geometric relationships among tokens in the output vocabulary is insufficient to explain subliminal transmission. The larger the model, the less tenable this "static geometry" narrative becomes.

Causal Control Is the Key Signal

To test genuine causal control, the researchers designed an elegant intervention experiment: they copied the transient state at the answer position of one number prompt to another number prompt at five different depths, then observed which prompt the final animal score followed.

The results were highly significant — donor-control AUC rose from 0.254 to 0.540, a paired change of +0.286 (95% CI [+0.272, +0.300]), with increases observed across all 18 concepts. Even more notably, this contrast held with as few as eight transformer blocks remaining, while specificity and identity controls remained small or precise.

This result demonstrates that hidden states causally participate in feature transmission — not merely as a correlational coincidence. Causal timing constitutes independent and stronger evidence than static geometry.

This experiment is essentially an Activation Patching technique — a standard method in mechanistic interpretability research for establishing causal attribution. The logic is: if replacing the hidden state at a given layer from a donor prompt into a recipient prompt changes the final output, then that layer's hidden state exerts causal control over the output. AUC (area under the curve) here measures whether, under a given intervention depth, the patched prompt's output score more closely resembles the donor's or the recipient's original distribution — AUC = 0.5 indicates chance level, AUC = 1 indicates complete alignment with the donor.

The significance of donor-control AUC rising from 0.254 to 0.540 is as follows: pre-intervention (0.254) is near-random or slightly below, suggesting limited baseline output differences between the two prompts; post-intervention (0.540) is significantly above chance, indicating that replacing hidden states genuinely shifts the animal feature output tendency — providing direct experimental evidence for a causal chain, rather than inferring it from static similarity in the output layer alone.

Multi-Token Measurement Exposes a "Length Confound"

The study also tested multi-token scenarios on two Qwen models, yielding a cautionary finding. When scoring each digit in a sequence token by token, the positive association observed in single-token settings could not be replicated.

Using per-token averaging, however, artificially creates a positive pooled association — but this association disappears once number width is controlled for. In other words, the apparent association is likely nothing more than a length confound, not genuine semantic entanglement.

This serves as a warning to researchers: in multi-token measurement settings, statistical artifacts must be treated with extreme caution, or misleading conclusions are easily drawn.

Conclusion: Constraints on Interpretation, but No Locked-Down Mechanism

The core contribution of this research is to clearly demonstrate that fixed geometry, observational readability, causal timing, and multi-token measurement are four distinct properties of this frozen prompting channel. Each constrains token-level explanations from a different angle — static geometry has limited explanatory power, length confounds must be eliminated, and causal control provides the strongest evidence.

However, the authors maintain a measured stance: these findings constrain token-level explanations without truly identifying the underlying mechanism of feature transfer during training. The complete causal chain behind subliminal learning remains an open research question.

For researchers focused on AI safety and interpretability, the methodological value of this paper may exceed its specific conclusions — it provides a reusable experimental paradigm for rigorously distinguishing "correlation" from "causation."

Background

Length Confound is a classic statistical artifact in multi-token evaluation. The token width of a number (i.e., how many tokens it is split into after tokenization) is not randomly distributed: larger numbers typically occupy more tokens, and different numeric ranges may co-occur systematically with different semantic domains in training data. When researchers score each token in an entire number sequence and then average the scores, the mean simultaneously absorbs the influence of "number length" as a confounding variable — longer numbers contribute more scoring points, and if longer numbers happen to have a statistical association with certain animals, the mean will be spuriously inflated.

The standard approach to controlling for number width is to include width as a covariate in a regression model, or to analyze separately within subgroups of equal width. This paper finds that once this variable is controlled for, the previously observed positive pooled association disappears — indicating that what seemed like a significant multi-token result was not evidence of genuine semantic entanglement. This is an important warning for researchers who rely on automated pipelines to evaluate language model outputs at scale.