AgentDojo
A security benchmark for prompt injection attacks against LLM agents. It contains real-world prompt injection attack samples and is used to evaluate the effectiveness of detectors and defense schemes.
Timeline (last 90 days)
AgentDojo contains 629 real injection attacks, each hidden in ordinary tool output.
In tests with 629 AgentDojo attacks and 97 benign tool outputs, jailbreak-detector-large achieved a capture rate of 51% with a false positive rate of 2%.
Prompt Guard 2 scored 0.999 on human-written injection instructions, but scored between 0.004 and 0.140 on real AgentDojo attacks, all below the default threshold of 0.5.
All Facts (3)
AgentDojo contains 629 real injection attacks, each hidden in ordinary tool output.
50%UnverifiedIn tests with 629 AgentDojo attacks and 97 benign tool outputs, jailbreak-detector-large achieved a capture rate of 51% with a false positive rate of 2%.
50%UnverifiedPrompt Guard 2 scored 0.999 on human-written injection instructions, but scored between 0.004 and 0.140 on real AgentDojo attacks, all below the default threshold of 0.5.
50%