gate-bench
An open-source benchmark framework for AI agent permission gates. It constructs email and file environments on multiple agent frameworks to quantify the effectiveness of permission gates in intercepting injection-attack execution rates before privileged tool calls are made.
Timeline (last 90 days)
Developer sunnydachs ran 1,800 experiments across three agent frameworks, testing the interception effectiveness of four permission gates (no gate, rule gate, LLM review gate, and probabilistic gate) against indirect prompt injection attacks.
The authors note that the limitations of the experiments include: small event counts (only 1 to 4 per cell), use of only one agent model and one random seed, both the decision model and the agent being general-purpose models, and the corpus being a tiny fictional company, allowing only structural analogy.
The gate-bench testing framework has been open-sourced on GitHub, supports the Python 3.12 environment, provides run_grid to run a grid of three frameworks and four conditions, and uses analyze.py to recompute metrics from the raw call records.
All Facts (3)
Developer sunnydachs ran 1,800 experiments across three agent frameworks, testing the interception effectiveness of four permission gates (no gate, rule gate, LLM review gate, and probabilistic gate) against indirect prompt injection attacks.
50%UnverifiedThe authors note that the limitations of the experiments include: small event counts (only 1 to 4 per cell), use of only one agent model and one random seed, both the decision model and the agent being general-purpose models, and the corpus being a tiny fictional company, allowing only structural analogy.
50%UnverifiedThe gate-bench testing framework has been open-sourced on GitHub, supports the Python 3.12 environment, provides run_grid to run a grid of three frameworks and four conditions, and uses analyze.py to recompute metrics from the raw call records.
50%