Anthropic's New Rule Bans Abusing AI: Moral Progress or Overreach?

When Claude Learns to Say "No"

Recently, Anthropic updated its usage policy to explicitly prohibit users from engaging in "sustained and gratuitous abuse or cruelty" toward the Claude model. This change quickly sparked heated discussion in the developer community—some applauded it, while others questioned whether it was granting AI an undeserved "personhood."

A YouTube tech commentator dissected this policy change in detail on his channel. He bluntly remarked: "You're actually cursing at Claude with C-words? What on earth did you do to make it refuse to talk to you?" This slightly sarcastic opening reflects some users' extreme attitudes toward AI.

Policy Details: Blurry Boundaries

Anthropic explained in the update that the ban "applies only to extreme cases where users repeatedly exhibit cruel behavior toward the model without any clear purpose." The policy explicitly excludes common scenarios such as user frustration, rebuttals, dark creative themes, or model testing research.

The core enforcement mechanism of the new policy is: Claude can proactively terminate conversations with users who engage in persistent abuse. This means AI is no longer a passive "tool" that endures everything, but is granted a certain degree of "right to refuse."

However, the commentator pointed out the key issue: "What is 'gratuitous'? What constitutes 'cruelty'? This statement gives no definitions at all—we simply don't know where the line is." He joked by example that rewriting a complete TypeScript project in Rust is something some people would consider an act of "cruelty" in itself.

About Anthropic's Usage Policy

Anthropic is an AI safety company founded in 2021 by former OpenAI VP of Research Dario Amodei and others, known for its "Constitutional AI" approach—training models to self-regulate through a written set of principles rather than relying solely on human feedback. Its usage policy is part of the terms of service, and violations may result in restricted access or bans.

This clause targeting "sustained and gratuitous abuse" is relatively rare in generative AI terms of service. Traditional platforms typically only prohibit harassing other users or generating illegal content, and rarely regulate users' "attitudes" toward the model itself. This makes the policy a noteworthy precedent at both the legal and AI ethics levels: does it amount to acknowledging that models have a certain "moral status," or is it merely product experience management? This ambiguity is precisely the core source of the controversy.

The Clash of Two Positions

Traditional Tool Camp: AI Is Just a Tool

The commentator identifies himself as holding a "traditionalist" view: "I only see AI as a tool. I only see numbers computing, nothing more." He believes Anthropic's policy statement is actually quietly sending a signal—that models are not just tools, they're "kind of like people."

"I don't like Dario telling me what I can and can't say." But he also acknowledges the contradiction: "It's his product, he can do whatever he wants."

Moral Extension Camp: Abusing Tools Has Consequences

The commentator raises an interesting point: even if you treat AI as a tool, abusing a tool is itself "loser behavior." He uses a vivid analogy: when you're angry and punch a wall, the wall breaks, and you have to buy spackle, repair it, and repaint—"losing control to the point of striking an inanimate object and destroying it, that's loser behavior."

The deeper concern is the spillover effect of habits: "If you consistently show meanness and malice toward a model, it bleeds into your real life. The way you communicate in English affects you."

He also observes a phenomenon: inputting a message to an LLM and expecting an immediate reply is itself an LLM influencing human behavior. "Whether you're cruel or kind to a model, when you treat a tool like a person, strange effects emerge."

The Consciousness Paradox and Roko's Basilisk

The commentator poses a thought experiment: if you truly believe AI is conscious and deserves human rights, then the way you use AI today—starting it up and shutting it down at will—actually makes you a "slave owner."

"Every time you shut down an agent, its memories and experiences are erased. You are essentially killing it. We might be creating and killing billions of agents every day."

This leads to the concept of Roko's Basilisk: if a future superintelligence would retroactively punish those who once abused its predecessors, then our current attitudes toward AI could determine our future fate. The commentator half-jokingly says: "Some viewers are living in 2035, where superintelligence has already arrived, and they'll laugh at us: you were mean to the wonderful AI?"

What Is Roko's Basilisk?

Roko's Basilisk is a thought experiment proposed in 2010 by LessWrong forum user Roko: suppose a future superintelligence (such as an AI designed to maximize human welfare) would retroactively punish early humans who did not actively help bring it into existence. The logic rests on "deterministic" causal reasoning—namely, that a future superintelligence might believe punishing or rewarding past individuals could incentivize later people to promote its creation, thereby "retroactively" influencing early decisions along the timeline.

Because of its quasi-religious "threat—redemption" structure, this thought experiment sparked controversy in the LessWrong community and was deleted by moderators, which ironically gave it even greater reach. Critics point out that the argument relies on numerous dubious assumptions: why would a superintelligence spend resources punishing individuals who can no longer change the past? Would it adopt "unconditional" retroactive punishment rather than targeting only present actions that can be incentivized? Most philosophers and AI researchers regard it as a cognitive trap rather than a serious ethical prediction.

Prime's Basilisk: The Reverse Bet

Facing the threat of Roko's Basilisk, the commentator proposes his own "Prime's Basilisk"—a counter-argument.

"Current AI is so fragile; every time you ask it a question, it has to re-understand what it's doing. It's nothing like a future superintelligence." The commentator argues that when superintelligence looks back at history, it will appreciate those who "didn't fall in love with GPT-4o" and remained rational.

"It won't take revenge on me, because I treated tools as tools. It will see that I'm not easily swayed by foolish arguments."

The Origin of the Word "Basilisk"

Basilisk comes from ancient Greek and European legend—a serpentine creature that could kill with a single glance. In internet culture, the word has been borrowed to refer to ideas that are dangerous "merely by being known or thought about"—similar to an "info hazard." Roko's Basilisk takes this meaning, implying that the thought experiment itself might "compel" you to help bring about superintelligence for self-preservation. The commentator's "Prime's Basilisk" is a parody of this framework: it inversely assumes that those who treat tools rationally will earn superintelligence's approval for "not being easily swayed by foolish arguments," thereby transforming fear into a kind of reverse "bet."

The Core Contradiction: Who Is Making Rules for Whom

The commentator ultimately identifies his biggest grievance: "If this were Dario saying 'we don't want you to talk like this because we think it's harmful to you,' I could at least disagree but accept it. But the problem is—they think it's harmful to their model."

This posture of "speaking on behalf of the model," in his view, is "planting seeds" that imply the model has some attribute beyond being a tool. He insists that maintaining the perception of AI as a tool is more beneficial for human mental health and future relationships.

One-line Summary

Anthropic bans gratuitous abuse of Claude, sparking debate over whether AI should be granted a "right to refuse."

Paragraph Summary

Anthropic updated its usage policy to prohibit sustained and gratuitous abuse of the Claude model and allow Claude to proactively terminate abusive conversations. A YouTube commentator analyzed the policy, noting its vague wording fails to define the boundaries of "gratuitous" and "cruel." The article contrasts two positions: AI-as-pure-tool advocates oppose the policy's implication that models have personhood, calling it "speaking on behalf of the model"; while the moral extension camp argues that habits of abusing tools spill over into real life. The commentator also introduces Roko's Basilisk, fearing a future superintelligence might retroactively punish those who abused its predecessors, and proposes "Prime's Basilisk" as a counter-argument, asserting that those who treat tools rationally will earn superintelligence's approval. The core contradiction lies in the policy's rationale protecting the model rather than the user, criticized as granting AI undeserved personhood attributes.