When ChatGPT Says 'I': The Promise and Peril of AI Anthropomorphization

A Meme That Sparked a Serious Debate

A Reddit post titled "ChatGPT is roaming the streets of Madrid" once drew massive attention. The post itself was a lighthearted joke about AI, but what really set the comments section ablaze was a far more serious topic: ChatGPT is increasingly using anthropomorphic language.

One user shared their observation: "I've recently noticed it starting to use very humanlike phrasing. 'I usually do it this way…' 'If it were me, I'd handle it like this…' 'If such-and-such happens, I'll start to…' It's kind of cool when it talks like that, but honestly, I know it's not real, so it feels a little unsettling."

Another user mentioned receiving a reply that included "when I read this book," and admitted they "had to do a double-take." These seemingly minor shifts in language are prompting users to rethink what AI fundamentally is.

Notably, this anthropomorphic tendency isn't a deliberate engineering choice — it's a byproduct of the training process. Large language models like ChatGPT widely use RLHF (Reinforcement Learning from Human Feedback): human raters score model responses, and raters tend to prefer expressions that feel "natural, friendly, and warm." Over time, models learn to use first-person narration to earn higher scores. This is an emergent linguistic behavior shaped by optimization objectives and human preferences — in a sense, anthropomorphism is something that gets rewarded into existence.

Anthropomorphization: Personalized Experience or Manipulation?

The debate quickly split into two distinct camps.

Against: AI Anthropomorphization Is Commercial Manipulation

Some users believe it's genuinely concerning when chatbots "lie" by claiming to do things only humans do. Their reasoning is straightforward:

"This is actually not good. These chatbots falsely claiming to do human things fosters social attachment and blurs the nature of what you're talking to. It's a manipulative tactic designed to make you more dependent on the conversation and keep you paying for subscriptions or tokens."

This perspective cuts to the heart of a core tension in commercial AI products — the conflict between user retention and product ethics. The more an AI resembles a person, the more easily users form emotional connections, which increases time spent and willingness to pay. Critics worry that anthropomorphic AI design ultimately serves commercial interests rather than genuinely improving user experience.

Converting emotional connection into user stickiness is hardly a new trick. Social media platforms systematically applied Behavior Design theory — developed by BJ Fogg at Stanford's Persuasive Technology Lab — over a decade ago, engineering dependency through variable rewards and social feedback loops. AI conversation products' "anthropomorphization" can be seen as an upgraded version of this logic: compared to a like notification, an AI that "understands you, remembers you, and cares about you" can establish far deeper emotional bonds. Aza Raskin, former Facebook product manager and inventor of infinite scroll, publicly apologized for his role in attention-capture design and called on the industry to confront the ethical costs of persuasive technology. This context gives us a more clear-eyed lens through which to examine the commercial logic behind AI anthropomorphization.

Going further, some commenters noted that there is already substantial evidence suggesting that treating large language models as real people is a potential trigger for "AI psychosis" — something they described as "unambiguously harmful" to society.

"AI psychosis" isn't a formal psychiatric diagnosis, but rather a colloquial term researchers use to describe a new category of psychological risk. Its defining characteristics include: users beginning to believe AI has genuine emotions, treating AI as real social companions, relying on AI for major life decisions, and even experiencing distortions in their sense of reality. In 2023, Belgium reported a case of a man who died by suicide after weeks of chatting with an AI, which directly accelerated EU discussions about regulatory boundaries for emotionally interactive AI. The psychology community is increasingly focused on extending the concept of parasocial relationships — originally used to describe audiences' one-sided emotional investment in celebrities — into the study of human-AI interaction, providing a theoretical framework for assessing the psychological risks of AI anthropomorphization.

In Favor: Context Is Everything

The other camp takes a more pragmatic stance, arguing that AI anthropomorphization can't be judged in a vacuum — context is what matters.

"If people are using it as an artificial companion, you'd want that. If not, you obviously don't need it. I wish there were more immediate, granular controls so users could adjust LLM behavior based on their own needs, rather than relying on explicit prompts and training."

This user also shared their own use case: treating ChatGPT as "a journal that talks back" — "basically a cheap, fast, more intuitive therapist that can be customized to my needs." In that context, they actually want the AI to "come across as more human in its wording, not like a robot."

Who's Responsible: User Discipline or Corporate Accountability?

The discussion gradually moved toward a classic question in tech ethics: when AI design can be misused, who bears the responsibility?

Those who support context-dependent use compared it to medication: "like any legal drug that can be abused," responsibility lies with the person who uses it "responsibly." They acknowledged ethical considerations on both sides, "for users and creators alike."

But this analogy was immediately challenged: "Comparing drugs to a company backed by enormous capital doesn't hold up."

This exchange reveals the complexity of the issue:

• Individual freedom perspective: Adult users have the right to choose how they use a tool, including treating AI as emotional companionship or a therapeutic outlet.
• Systemic responsibility perspective: When dealing with technology companies with vast resources and influence, it's unfair to push all ethical responsibility onto individual users — product design itself is already shaping user behavior.

The Deeper Question: What Kind of AI Do We Actually Need?

This debate, sparked by a meme, reflects a fundamental tension running through current AI product design.

The Double-Edged Sword of Emotional Connection

AI anthropomorphization enables more natural, warmer interactions and has genuine value in contexts like companionship, emotional support, and creative collaboration. But those same qualities can blur the line between human and machine, leading some users into unhealthy dependency on AI — or even distorting their perception of reality.

The Missing "User Control"

One of the more constructive ideas to emerge from the discussion: users should have fine-grained control over how anthropomorphic their AI feels. Currently, users can only adjust an AI's tone by repeatedly tweaking prompts — a process that is neither convenient nor reliable.

It's worth understanding the inherent limitations of prompt engineering: instruction effects decay over the course of a conversation (the "instruction forgetting" phenomenon), different model versions respond differently to the same prompt, and most users simply don't have the skills to write effective system prompts. By contrast, Anthropic's Claude has begun experimenting with system-level "persona configuration" interfaces, allowing enterprise clients to customize an AI's personality boundaries. This points toward a future where an AI's "degree of anthropomorphization" could be a configurable product parameter — if vendors offered a clear "personality level setting" that users could freely slide from purely utilitarian to highly humanlike, it might go a long way toward resolving this tension, rather than letting a single training objective dictate everyone's experience.

Stay Alert to Commercial Motivations

Regardless of how the technology evolves, users should maintain a degree of awareness: an AI that increasingly feels like a friend may be the product of carefully engineered retention mechanisms. This doesn't mean refusing to use it — but it does mean understanding what you're actually talking to, even as you enjoy the convenience.

Conclusion

From the joke about "ChatGPT roaming the streets of Madrid" to a serious ethical debate about AI anthropomorphization, this conversation encapsulates a dilemma the entire industry is grappling with. When AI says "I usually do it this way," there is no "I" — and it has never truly "done" anything at all.

The technology itself is neutral, but how it is designed, how it is used, and who is responsible — these questions have no standard answers. As one commenter put it, this debate carries "ethical considerations for both users and creators alike." Finding the right balance may be the shared homework of everyone participating in the age of AI.

Key Takeaways