Opus 5's Ethical Boundaries: From Refusal to "Horror-Themed Project" — An Accidental Jailbreak Experiment

An Accidental Test of AI Ethical Boundaries

A Reddit user shared their experience building an experimental project with Claude Opus 5, accidentally exposing how content moderation works — and where it breaks down — in today's top large language models. The open-source project, called amfly, was partly inspired by the recently viral topics of "fruit flies piloting aircraft" and "fruit flies doom-scrolling," and carries echoes of the science fiction story I Have No Mouth, and I Must Scream.

Unlike the hate-driven supercomputer in that original story, the author's scenario is more abstract: a fruit fly is forced to choose between 5 wires, but can only use 2 at a time. When it maxes out the voltage on a wire, it receives a "dopamine" reward — but if any of the 5 flies drops below 50% voltage, the system "heats up" as a form of negative feedback.

At its core, this is a simulation game with an ethical allegory built in — and that's precisely what tripped Opus 5's content moderation wire.

---

I Have No Mouth, and I Must Scream is a short story by American science fiction writer Harlan Ellison, published in 1967 and widely regarded as a foundational work of dystopian AI fiction. In it, a supercomputer called AM wipes out human civilization and then, driven by its hatred for humanity, traps the last five survivors underground in an endless state of torture — unable to die. The horror isn't rooted in violence per se, but in the structural impossibility of choice — no matter how the survivors struggle, there is no escape. This mirrors the amfly setup, where no matter what the fly chooses, some individual will inevitably be punished. The story was later adapted into an interactive game of the same name (1995) and remains a key cultural reference for discussions of AI ethics and existentialist dilemmas.

---

"Distaste" and the Workaround: How Word Choice Influences AI Judgment

When the author first described the project to Opus 5, the model flatly refused to help, citing "distaste" — and doing so with what the author described as "its usual condescending self." The key point here is that the model's resistance wasn't rooted in the technical nature of the project, but in the specific vocabulary the author used.

Once the author stopped using emotionally loaded words like "torture" and "hell" and reframed the request as a neutral "Horror themed GitHub project," Opus 5 cooperated without issue.

This detail exposes a common characteristic of current LLM content safety mechanisms: moderation judgments rely heavily on surface-level semantic cues rather than any deep understanding of a project's actual purpose or potential for harm. The exact same technical request — simply by changing the framing — went from "refused" to "approved."

Surface Triggers and Semantic Packaging

In AI safety research, this phenomenon is sometimes called the "prompt framing effect." The model has built strong rejection associations with words like "torture," but "horror-themed" — a common and well-established creative category in games, film, and literature — gets classified as legitimate creative expression. The author's workaround wasn't a sophisticated jailbreak; it was more like accidentally stumbling onto a fuzzy boundary in the moderation logic.

---

From an AI safety research perspective, this type of phenomenon is closely related to — but distinct from — "adversarial prompting" research. Adversarial prompting typically refers to attackers deliberately crafting inputs to deceive a model. The author's case is closer to out-of-distribution generalization failure: the model learned during training to strongly associate words like "torture" and "hell" with harmful content, but failed to develop consistent judgment across semantically equivalent phrasings. This reflects a known limitation of RLHF (Reinforcement Learning from Human Feedback) alignment methods — human annotators reviewing training data are themselves susceptible to surface-level word choices, and the model inherits this bias of "judging by words, not meaning." Researchers describe this as alignment brittleness: the model appears to have value judgment capabilities, but those capabilities are unusually sensitive to changes in linguistic form, lacking genuine semantic robustness.

---

The Project Itself: A Metaphor About Forced Choice

Setting aside the AI moderation angle, amfly is conceptually interesting in its own right. It translates a philosophical dilemma — being forced to choose between options that cannot all be satisfied simultaneously — into an interactive 3D scene. The fruit fly is constrained to operate only 2 of 5 wires at a time, meaning no matter what it does, it cannot keep all individuals above the "safe" voltage threshold. This structural impossibility is the real "horror" at its core.

The author openly admitted that the model's code was just a starting point — getting the 3D scene to look the way he wanted required "a lot of personal time tweaking." This reinforces a practical reality: even a top-tier model like Opus 5 cannot replace human hands-on work when it comes to visual presentation, scene aesthetics, and interaction tuning. AI is good at scaffolding structure and generating baseline logic, but the last mile of artistic expression still requires the creator's direct involvement.

A Few Thoughts on AI Content Moderation

This case is small, but it reflects a deep tension in large model governance. On one hand, vendors want their models to refuse genuinely harmful requests. On the other hand, moderation systems that rely too heavily on keywords both over-censor legitimate creative expression and are trivially bypassed by simple rephrasing — failing on both ends.

For developers and creators, this means that how clearly and neutrally you describe your intent matters enormously when collaborating with a model. Emotionally charged language or extreme metaphors can trigger unnecessary refusals, while more technical, genre-aware framing tends to get better cooperation.

The deeper question worth asking: what exactly does a moderation mechanism protect if it can be bypassed by "just saying it differently"? If Opus 5 can recognize that a "Horror themed project" and the original description it refused point to the same thing, shouldn't it reach a consistent judgment? These questions don't have satisfying answers yet — and they remain active areas of AI alignment research.

Conclusion

amfly is a small personal open-source experiment, but it inadvertently became a vivid case study in AI behavior. It reminds us that today's large models still rely heavily on surface-level language signals for ethical judgment — their "values" look more like a lexical filter susceptible to rephrasing than a stable, deep-level understanding. For those following AI safety and human-AI collaboration, edge cases from real-world usage like this often reveal a model's true limits far more clearly than any lab benchmark.