Unverified50% confidenceOpinionExact time
Language models cannot reliably distinguish between "data" and "instructions," so safety boundaries built solely on prompts are inherently fragile.
1
Sources
50%
Confidence
Long-term
Relevance
10/17/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe safety behavior of large language models is a statistically learned tendency rather than hard-coded rules, and cannot provide mathematical safety guarantees.79% similarUnverifiedLarge models lack a natural mechanism for distinguishing 'instruction source trustworthiness' when processing context, making it difficult to automatically identify which text constitutes authorized executable instructions and which is merely data content to be processed.77% similarUnverifiedDefending against prompt injection should rely on structural constraints rather than expecting the model to identify attacks on its own, as semantic-layer detection has been proven to be extremely unreliable.76% similarUnverifiedRelying on prompt constraints to prevent dangerous operations is inherently unreliable, because the model may be jailbroken, make misjudgments, or forget constraints during long conversations.73% similarUnverifiedLarge language models are fundamentally unable to distinguish between system instructions and user input at a structural level, making prompt injection attacks technically impossible to fully eliminate.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1055032API
curl https://kongchang.com/api/v1/knowledge/claims/1055032MCP
get_claim(id=1055032)