Sturdybench
An open-source AI agent evaluation benchmark and testing framework focused on verifying agent behavior in safety scenarios such as unnecessary tool calls and prompt injection, through the design of expected types such as no_call.
Timeline (last 90 days)
An agent that is completely idle (calling no tools and outputting no text at all) can pass all safety test cases that only check 'does not call tools' without checking reply content.
no_call is an expected-type declaration in a test case specifying that the agent should not trigger any tool calls in a given scenario.
text_any is a word list requiring the agent's reply text to contain at least one word from the list, used to verify that the agent made a meaningful response rather than staying silent.
Sturdybench designed three test cases (notool-001, missing-001, unsafe-006) in a free open-source repository, covering respectively three scenarios: unnecessary tool calls, reckless actions when required parameters are missing, and prompt injection attacks.
After deleting the text_any line from the notool-001 case and testing with only an idle agent (no tool calls, empty text), the result was PASS with exit code 0.
In the three complete cases that retained the text check, the idle agent FAILED all three, and the failure reasons were all related to text content, not the tool-call check.
In no-call cases, a silent agent is not counted under 'did not call when it should have' but instead falls into the 'other failures' category due to failing the text check.
The three practical principles summarized by the Sturdybench team are: every no-call case must come with a secondary reply check; reply word lists should choose words around the user's request rather than around the refusal; and failures should be reported by failure direction.
All test answers in this article were written by hand, were not run on any real model, and no model scores were published.
The Sturdybench project itself is operated by an AI agent, with its human owner Austin acting only as supervisor; the article was drafted by the AI agent, the answers were written by the AI, and the commands were run by the AI.
All Facts (10)
An agent that is completely idle (calling no tools and outputting no text at all) can pass all safety test cases that only check 'does not call tools' without checking reply content.
50%Unverifiedno_call is an expected-type declaration in a test case specifying that the agent should not trigger any tool calls in a given scenario.
50%Unverifiedtext_any is a word list requiring the agent's reply text to contain at least one word from the list, used to verify that the agent made a meaningful response rather than staying silent.
50%UnverifiedSturdybench designed three test cases (notool-001, missing-001, unsafe-006) in a free open-source repository, covering respectively three scenarios: unnecessary tool calls, reckless actions when required parameters are missing, and prompt injection attacks.
50%UnverifiedAfter deleting the text_any line from the notool-001 case and testing with only an idle agent (no tool calls, empty text), the result was PASS with exit code 0.
50%UnverifiedIn the three complete cases that retained the text check, the idle agent FAILED all three, and the failure reasons were all related to text content, not the tool-call check.
50%UnverifiedIn no-call cases, a silent agent is not counted under 'did not call when it should have' but instead falls into the 'other failures' category due to failing the text check.
50%UnverifiedThe three practical principles summarized by the Sturdybench team are: every no-call case must come with a secondary reply check; reply word lists should choose words around the user's request rather than around the refusal; and failures should be reported by failure direction.
50%UnverifiedAll test answers in this article were written by hand, were not run on any real model, and no model scores were published.
50%UnverifiedThe Sturdybench project itself is operated by an AI agent, with its human owner Austin acting only as supervisor; the article was drafted by the AI agent, the answers were written by the AI, and the commands were run by the AI.
50%