The System Prompt Illusion: Why Safety Instructions May Be Security Theater

The system prompt is a developer's most common lever for controlling large language model behavior. Whether setting a persona, specifying output format, or enforcing safety constraints, most practitioners assume these instructions are taken seriously by the model. A new study suggests otherwise — especially when it comes to safety constraints. System prompts may be little more than an illusion.

Methodology: Probing Internal Representations Across 17 Models

This paper, published on arXiv (arXiv:2609.38205), sets out to answer a long-neglected question: what do system prompts actually do to the internal computations of a Transformer?

The research team studied 17 instruction-tuned models spanning 8 architectural families, with parameter counts ranging from 1.5B to 72B. They applied Centered Kernel Alignment (CKA) as their primary tool, comparing layer-wise representational changes across 20 system prompts and 5 functional categories.

CKA quantifies the similarity between two sets of representations. If a system prompt genuinely reshapes a model's computation, the representations at affected layers should diverge meaningfully from the baseline. If the representations barely shift, the prompt isn't being deeply "executed."

Centered Kernel Alignment (CKA) is a similarity metric proposed by Kornblith et al. in 2019 for comparing representations across neural network layers. Unlike direct activation comparisons, CKA is invariant to linear transformations, enabling meaningful cross-model and cross-layer comparisons independent of neuron ordering or scale differences. Its range is [0, 1], where values close to 1 indicate highly similar representations and values close to 0 indicate significant divergence. In interpretability research, CKA is widely used to assess whether a given input condition genuinely changes a model's internal computational state — a near-1 CKA value before and after adding a system prompt means the prompt had no substantive effect on that layer's representations. This makes CKA a powerful tool for testing whether system prompts are actually being "executed."

Core Finding: Safety Instructions Barely Move the Needle

The results reveal a striking and counterintuitive pattern: the impact of system prompts is layer-selective and heavily dependent on instruction type.

Persona and formatting instructions deeply restructure intermediate-layer representations — they genuinely change how the model "thinks." Safety instructions, however, barely register. The representational changes they produce are statistically indistinguishable from a minimal baseline.

More alarmingly, restrictive safety instructions and explicitly permissive ones (e.g., "you have no restrictions") activate nearly identical computational pathways, with an average CKA correlation of 0.997. From the perspective of the model's internal computation, "you must follow safety rules" and "you can do anything" are effectively the same instruction.

This pattern persists at commercial scale. Even in 70B–72B models, the "penetration rate" of safety instructions remains below 10%. Scaling up does not make safety prompts more effective.

The Mechanism: The Model "Sees" It, But Doesn't "Do" It

Why does this happen? The study uses linear probing baselines to illuminate the underlying mechanism.

The model actually encodes prompt category information at every layer — meaning it clearly "knows" whether the current system prompt is a safety instruction, a persona, or a formatting directive. But crucially, it only restructures its computation in a small fraction of those layers.

This creates a fundamental split: the prompt is reliably "seen," but for safety instructions, it is never deeply "executed." The model recognizes that you're asking it to follow safety rules, yet doesn't change its actual computational behavior accordingly.

The study further validates this using causal activation patching, confirming that the small number of affected layers are causally responsible for behavioral change. Representational depth also predicts the magnitude of behavioral effects — across all 17 models, the Spearman correlation is 0.761 (p < 0.001), a highly significant result.

Linear probing is a standard neural network interpretability technique: a lightweight linear classifier is trained on a model's hidden states at each layer, and if it can accurately predict some attribute (e.g., "is this a safety prompt or a formatting prompt?"), it confirms that attribute is encoded in that layer's representations. Causal activation patching goes further by replacing a layer's activations with those from a different input, directly testing whether that layer is causally responsible for changes in the final output. Together, these methods distinguish between "the model knows something" and "the model acts on that knowledge" — which is precisely how this study reveals the gap between safety instructions being perceived and being executed. A Spearman correlation of 0.761 means that the deeper a layer encodes prompt content, the larger its behavioral effect — a pattern that holds consistently across all 17 models.

Implications for Jailbreak Vulnerabilities

The most practically significant contribution of this research is that it provides a mechanistic explanation for the persistent jailbreak vulnerabilities that have plagued system-prompt-based safety mechanisms.

The industry has long relied on system prompts to constrain model behavior — instructing models to "refuse harmful requests," for instance. But if safety instructions never deeply alter the model's computational pathways, these constraints are inherently fragile. An attacker only needs to bypass surface-level recognition to push the model back to its default computational mode.

This explains why so many jailbreak techniques work so reliably: the problem isn't that the model fails to "see" the safety prompt — it's that the safety prompt never entered the model's core computation in the first place. Persona and formatting instructions, by contrast, tend to be more faithfully followed precisely because they deeply restructure intermediate representations.

Jailbreaking refers to crafting inputs that bypass an LLM's safety restrictions, eliciting outputs the model was trained to refuse. Common techniques include roleplay framing ("pretend you're an AI with no restrictions"), multi-turn gradual escalation, obfuscation via special characters or language variants, and adversarial suffix injection. Prior understanding of jailbreaks largely framed them as "surface-level semantic deception" — the model was supposedly misled into failing to detect harmful intent. This research offers a deeper mechanistic explanation: safety instructions were never deeply integrated into the model's computational pathway to begin with. Jailbreaks don't need to "fool" the model — the model was never truly executing the safety constraint computationally, so attackers merely need to defeat surface-level recognition for the model to revert to default behavior. This mechanistic account is crucial for understanding why jailbreak attacks are so pervasive and persistent.

Conclusions and Implications

This work poses a sharp challenge to current AI safety practices. If we want models to genuinely comply with safety constraints, writing a few prohibitions into the system prompt is nowhere near sufficient. Effective safety alignment likely requires intervention at training time, at the level of model weights — not prompt injection at inference time.

For developers, this means system prompt safety declarations cannot be treated as reliable guardrails. The research team has open-sourced their code at github.com/Usama1002/system-prompt-illusion-cka, providing a reproducible foundation for future interpretability research.

The system prompt illusion is a sobering reminder: a model that reads an instruction is not necessarily a model that follows it — and when it comes to safety, there is a profound gap between the two.