Prompt Injection Attack

A prompt injection attack is a security attack targeting large language models (LLMs) and their applications. Attackers embed carefully crafted malicious instructions within input content to interfere with or override the model's original system prompt, manipulating the model into executing unintended behaviors such as bypassing safety restrictions, leaking sensitive information, or performing harmful operations. This attack exploits the inherent limitation of models in distinguishing normal user input from malicious instructions.