Attention Dilution
A phenomenon in the Transformer architecture when processing long sequences, in which attention weights become dispersed as context length increases, causing earlier information (such as instructions in the system prompt) to be marginalized in subsequent reasoning.
Core Facts
The constraining power of the system prompt is not absolute; as the number of conversation turns increases, the attention share received by early tokens (including the system prompt) in the Transformer's self-attention mechanism diminishes due to attention dilution, which is the technical root cause of the persona drift phenomenon.
Even models with million-token context windows like Gemini 2.5 face attention dilution in long-running complex tasks — as the context continues to grow, the model's attention to early information drops significantly.
Timeline (last 90 days)
The constraining power of the system prompt is not absolute; as the number of conversation turns increases, the attention share received by early tokens (including the system prompt) in the Transformer's self-attention mechanism diminishes due to attention dilution, which is the technical root cause of the persona drift phenomenon.