Prompt Injection Attack
A security attack targeting large language models in which an attacker embeds text disguised as system instructions within user input or external content, overriding or tampering with the original system prompt to hijack model behavior. The technique is analogous to traditional SQL injection attacks.
Core Facts
Prompt注入（Prompt Injection）是针对大语言模型应用的安全威胁，攻击者通过在输入中嵌入恶意指令试图覆盖系统原有行为约束
Indirect prompt injection attacks plant hidden instructions within normal business data to induce Agents into executing unauthorized operations, representing a security risk at the semantic level rather than the code level.
Red-teaming originated in the military and cybersecurity fields and, in the AI safety context, has developed into multiple layers including prompt-level attacks, training infrastructure security testing, and model capability risk testing.
Timeline (last 90 days)
Customer messages should be treated as task data rather than as authorization to modify workflows; this is a direct defense against prompt injection attacks.
Approaches to countering prompt injection attacks include enforcing human approval for sensitive operations, limiting agent permission scopes, and sandboxing execution environments, but no perfect solution currently exists.
A prompt injection attack refers to a situation where maliciously crafted web pages in an agent's environment inject new instructions, causing the agent's behavior to deviate from the designer's intentions.
Indirect injection refers to attackers hiding malicious instructions in external content that the model will read, such as webpage body text, email body text, or document content.
Because large models themselves cannot strictly distinguish between instructions and data, prompt injection is fundamentally very difficult to completely eliminate.
Large language models tend to solve problems using general reasoning while lacking strict adherence to domain-specific rules and self-verification capabilities.
Red-teaming originated in the military and cybersecurity fields and, in the AI safety context, has developed into multiple layers including prompt-level attacks, training infrastructure security testing, and model capability risk testing.
提示注入攻击（Prompt Injection）指恶意第三方在Agent处理的内容中嵌入指令，劫持Agent的行为意图
All Facts (12)
Prompt注入（Prompt Injection）是针对大语言模型应用的安全威胁，攻击者通过在输入中嵌入恶意指令试图覆盖系统原有行为约束
90%VerifiedIndirect prompt injection attacks plant hidden instructions within normal business data to induce Agents into executing unauthorized operations, representing a security risk at the semantic level rather than the code level.
80%VerifiedRed-teaming originated in the military and cybersecurity fields and, in the AI safety context, has developed into multiple layers including prompt-level attacks, training infrastructure security testing, and model capability risk testing.
75%VerifiedIndirect injection refers to attackers hiding malicious instructions in external content that the model will read, such as webpage body text, email body text, or document content.
65%VerifiedBecause large models themselves cannot strictly distinguish between instructions and data, prompt injection is fundamentally very difficult to completely eliminate.
65%VerifiedLarge language models tend to solve problems using general reasoning while lacking strict adherence to domain-specific rules and self-verification capabilities.
65%VerifiedThe core security risks faced by AI desktop agents include prompt injection attacks, privilege escalation risks, and irreversible operation risks.
65%UnverifiedCustomer messages should be treated as task data rather than as authorization to modify workflows; this is a direct defense against prompt injection attacks.
50%UnverifiedApproaches to countering prompt injection attacks include enforcing human approval for sensitive operations, limiting agent permission scopes, and sandboxing execution environments, but no perfect solution currently exists.
50%UnverifiedA prompt injection attack refers to a situation where maliciously crafted web pages in an agent's environment inject new instructions, causing the agent's behavior to deviate from the designer's intentions.
50%Unverified提示注入攻击（Prompt Injection）指恶意第三方在Agent处理的内容中嵌入指令，劫持Agent的行为意图
50%Unverified计算机使用能力的核心安全风险包括Prompt Injection攻击和未经授权访问敏感本地数据
50%