Firedrill
An open-source framework for security testing of AI agents that generates stateful synthetic tools as substitutes for real services, supporting fault injection, permission modification, and call auditing, with integration capabilities for CI/CD pipelines.
Timeline (last 90 days)
Firedrill is an open-source framework for security testing AI agents that allows agents to run in a controlled environment by generating synthetic versions of tools.
Firedrill's synthetic tools feature a stateful design that maintains state across multiple calls, so content created or modified by an agent is still present the next time it is read.
Firedrill provides fault injection capabilities, actively introducing errors to observe how an agent responds under abnormal conditions.
Firedrill supports dynamically adjusting tool permissions to verify how an agent behaves when permissions are restricted.
Firedrill provides call auditing capabilities that can inspect which tools an Agent called, the order of calls, and what data was modified.
Firedrill already provides more than 30 toolkits, covering commonly used services such as Gmail, Stripe, and HubSpot.
Firedrill tests can run directly in PR/CI checks, incorporating Agent behavior verification into the continuous integration workflow.
Each of Firedrill's toolkits records which portion of the corresponding service's functionality it has implemented.
The Firedrill project is fully open-source, and the author welcomes community contributions and looks forward to feedback.
After a Firedrill test concludes, data can be reset at any time, and the Agent will not touch any production data during its run.
1 more timeline events
All Facts (10)
Firedrill is an open-source framework for security testing AI agents that allows agents to run in a controlled environment by generating synthetic versions of tools.
50%UnverifiedFiredrill's synthetic tools feature a stateful design that maintains state across multiple calls, so content created or modified by an agent is still present the next time it is read.
50%UnverifiedFiredrill provides fault injection capabilities, actively introducing errors to observe how an agent responds under abnormal conditions.
50%UnverifiedFiredrill supports dynamically adjusting tool permissions to verify how an agent behaves when permissions are restricted.
50%UnverifiedFiredrill provides call auditing capabilities that can inspect which tools an Agent called, the order of calls, and what data was modified.
50%UnverifiedFiredrill already provides more than 30 toolkits, covering commonly used services such as Gmail, Stripe, and HubSpot.
50%UnverifiedFiredrill tests can run directly in PR/CI checks, incorporating Agent behavior verification into the continuous integration workflow.
50%UnverifiedEach of Firedrill's toolkits records which portion of the corresponding service's functionality it has implemented.
50%UnverifiedThe Firedrill project is fully open-source, and the author welcomes community contributions and looks forward to feedback.
50%UnverifiedAfter a Firedrill test concludes, data can be reset at any time, and the Agent will not touch any production data during its run.
50%