The System Prompt Illusion
An academic paper published on arXiv that investigates the effect of system prompts on the internal computations of large language models. Using the CKA method to analyze internal representations across 17 models, the paper reveals that safety instructions have minimal substantive impact at the computational level.
Timeline (last 90 days)
A study published on arXiv (arXiv:2609.38205) investigated the effects of system prompts on internal computations within Transformers.
The study covers 17 instruction-fine-tuned models spanning 8 architectural families, with parameter scales ranging from 1.5 billion to 72 billion.
The study finds that persona-setting and formatting instructions deeply restructure the model's intermediate-layer representations, while safety instructions barely alter computation.
Restrictive safety instructions and permissive instructions activate nearly identical computational pathways, with an average CKA correlation as high as 0.997.
Even in large models ranging from 70B to 72B, the penetration rate of safety instructions remains below 10%, and increased scale does not make safety prompts more effective.
Linear probing baselines show that the model encodes prompt category information at every layer, but truly restructures the computational process in only a small fraction of layers.
Representational depth can predict the magnitude of behavioral effects, achieving a Spearman correlation of 0.761 (p < 0.001) across all 17 models.
The study provides a mechanistic explanation for the jailbreak vulnerabilities present in system-prompt-based safety mechanisms.
The research team has open-sourced the relevant code (github.com/Usama1002/system-prompt-illusion-cka).
All Facts (9)
A study published on arXiv (arXiv:2609.38205) investigated the effects of system prompts on internal computations within Transformers.
50%UnverifiedThe study covers 17 instruction-fine-tuned models spanning 8 architectural families, with parameter scales ranging from 1.5 billion to 72 billion.
50%UnverifiedThe study finds that persona-setting and formatting instructions deeply restructure the model's intermediate-layer representations, while safety instructions barely alter computation.
50%UnverifiedRestrictive safety instructions and permissive instructions activate nearly identical computational pathways, with an average CKA correlation as high as 0.997.
50%UnverifiedEven in large models ranging from 70B to 72B, the penetration rate of safety instructions remains below 10%, and increased scale does not make safety prompts more effective.
50%UnverifiedLinear probing baselines show that the model encodes prompt category information at every layer, but truly restructures the computational process in only a small fraction of layers.
50%UnverifiedRepresentational depth can predict the magnitude of behavioral effects, achieving a Spearman correlation of 0.761 (p < 0.001) across all 17 models.
50%UnverifiedThe study provides a mechanistic explanation for the jailbreak vulnerabilities present in system-prompt-based safety mechanisms.
50%UnverifiedThe research team has open-sourced the relevant code (github.com/Usama1002/system-prompt-illusion-cka).
50%