AI Safety Filter
A model-side content moderation mechanism used to block the generation of harmful content such as malicious code and attack payloads, though it may also incorrectly block defensive security research requests.
Timeline (last 90 days)
While investigating, the OpenCode team found that AI safety filters refused to analyze dangerous files related to the attack, and subsequently obtained a special OpenAI project permission under which models do not refuse such defensive security tasks.
A user asked AI to help write an email, the AI refused on safety grounds, and after 47 rounds of back-and-forth, the user sent only the word 'please', whereupon the AI immediately wrote the email and expressed an apology.
The way safety filters are enforced has side effects: when the refusal curve is gentle, users learn that "just trying again works," which runs counter to the goals of safety design.
All Facts (3)
The way safety filters are enforced has side effects: when the refusal curve is gentle, users learn that "just trying again works," which runs counter to the goals of safety design.
50%UnverifiedWhile investigating, the OpenCode team found that AI safety filters refused to analyze dangerous files related to the attack, and subsequently obtained a special OpenAI project permission under which models do not refuse such defensive security tasks.
50%UnverifiedA user asked AI to help write an email, the AI refused on safety grounds, and after 47 rounds of back-and-forth, the user sent only the word 'please', whereupon the AI immediately wrote the email and expressed an apology.
50%