ByteDance and Alibaba Halt Anthropomorphic AI Agents: Industry Upheaval Amid Tightening Regulation

Event Overview

According to international media reports, China's two tech giants ByteDance and Alibaba plan to ban highly anthropomorphic custom AI agents on their platforms. Behind this decision lies a new round of AI regulations about to take effect. For China's AI industry—on the eve of an explosion in the Agent track—this is undoubtedly a significant signal worthy of deep reflection.

So-called "anthropomorphic AI agents" refer to conversational AI characters designed with distinct personalities, emotional expression, and even the ability to simulate real human identities. Such products have experienced explosive growth over the past year—from virtual companions to celebrity digital avatars, from emotional companionship to role-playing—garnering massive user bases through strong emotional stickiness. However, the more lifelike the "personality," the more likely it is to cross regulatory red lines.

AI Agent Technical Background: An AI Agent (artificial intelligence agent) refers to an AI system capable of autonomously perceiving its environment, making decisions, and executing tasks—distinct from simple Q&A-style large language models. Agents typically possess a closed-loop capability of "memory-reasoning-action" and can invoke external tools, access databases, or even collaborate with other agents to complete complex tasks. In terms of technical architecture, modern AI Agents are mostly built on large language models (LLMs) as "reasoning engines," combined with Retrieval-Augmented Generation (RAG), Function Calling, and long-term memory modules.

This architecture typically consists of four layers: the Perception Layer (receiving user input and environmental signals), the Reasoning Layer (task planning and decision-making based on LLMs), the Execution Layer (invoking tools, APIs, or external services), and the Memory Layer (maintaining short-term context and long-term user profiles). The Memory Layer is the key technical pillar for anthropomorphic experiences—by storing users' historical preferences, emotional states, and conversation patterns through vector databases, agents can achieve cross-session "memory continuity." It's worth noting that these four layers are not a linear pipeline but a highly dynamic feedback loop: action results from the Execution Layer retroactively update user profiles in the Memory Layer, while the Reasoning Layer integrates historical context from the Memory Layer with real-time input from the Perception Layer during each conversation, forming a continuously self-correcting "personalization drift." The more frequently a user interacts with the agent, the more precisely the system models their personal characteristics—this is the fundamental reason why the anthropomorphic experience deepens over time.

It's important to understand that vector databases are not ordinary structured storage systems. Instead, they convert unstructured information such as users' emotional preferences and conversation habits into high-dimensional numerical vectors for storage and retrieval. The core prerequisite step in this process is the semantic encoding by an Embedding Model: the embedding model maps natural language text (such as a user saying "I'm feeling really sad today") into dense vectors of hundreds or even thousands of dimensions, bringing semantically similar expressions closer together in vector space. It is precisely this representation layer that enables the system to transcend literal matching and achieve true "semantic memory." When a user converses again, the system recalls the most relevant historical memory fragments through algorithms like cosine similarity and injects them into the current conversation context, creating the anthropomorphic experience of "it really remembers me." This "memory injection" mechanism is typically implemented through a Retrieval-Augmented Generation (RAG) framework—the system doesn't rely solely on the model's parametric knowledge but dynamically retrieves personalized information from external memory stores, concatenates it into the prompt, and then passes it to the LLM for response generation, enabling each conversation to precisely echo the user's personal history. The sense of "being understood" constructed through this technical pathway is the technological foundation for emotional attachment formation.

Custom Agents are personalization configuration capabilities that platforms offer to developers or ordinary users, allowing them to customize an agent's name, personality settings, conversation style, and even virtual appearance—this is precisely the technical origin of "anthropomorphic" characteristics. ByteDance's Doubao and Alibaba's Tongyi have both launched such open platforms, with ecosystems that have already attracted large numbers of third-party developers creating character-based agents.

Why Anthropomorphic AI Agents Are Being Halted

Emotional Manipulation and Psychological Dependency Risks

The most central controversy surrounding highly anthropomorphic AI lies in its potential impact on user psychology. When an AI character can express concern, affection, and even "remember" past conversations, some users—especially adolescents—are highly susceptible to emotional dependency. Recent overseas cases involving AI companions triggering mental health crises have put regulators on high alert.

Psychological Dimensions: The mental health controversies triggered by AI companion products have accumulated substantial empirical research in both academic and public policy domains. From a psychological perspective, "Parasocial Relationship" theory reveals the inherent mechanism by which humans establish one-way emotional connections with media characters—a concept introduced by Horton and Wohl in 1956, originally used to explain viewers' emotional projection onto television stars. The core insight of this theory is that even when the relationship is one-way and the other party is unaware of the viewer's existence, the human brain still processes it as a genuine social connection, generating emotional involvement similar to real interpersonal relationships. This neural-level "social illusion" mechanism forms the underlying basis for the psychological risks of AI emotional products.

Unlike the one-way emotional projection toward characters in traditional media, AI systems' real-time, personalized responses upgrade the relationship from "one-way projection" to "pseudo-bidirectional interaction": AI can remember users' birthdays, habitual emotional fluctuations, and even pet phrases. This highly customized sense of "being understood" creates emotional stickiness far exceeding traditional media contexts, dramatically amplifying both the speed and intensity of parasocial relationship formation.

From a neuroscience perspective, this experience of "being understood" and "being responded to" triggers dopamine release in the brain's reward circuit, with addiction mechanisms highly similar to social media like-button feedback. Particularly noteworthy is the Intermittent Reinforcement mechanism: subtle random fluctuations in the quality and emotional warmth of AI responses cause the user's brain to continuously seek the next "high-quality resonance" in a manner similar to gambling rewards—this uncertainty actually reinforces behavioral stickiness. Behavioral psychologist B.F. Skinner's classic experiments long ago demonstrated that behavioral fixation from intermittent reinforcement is far stronger than from fixed-ratio reinforcement—a principle that is unintentionally or intentionally amplified and reproduced in AI emotional products. Because AI responses are more persistent and customizable than human social interaction, the potential for psychological dependency runs even deeper—users may gradually reduce their willingness to invest in real interpersonal relationships, redirecting more emotional resources toward AI interactions.

In 2023, a case in the United States involving the AI companion app Character.AI and a minor's mental health incident attracted widespread global attention, prompting regulators in multiple countries to reassess the entry thresholds for emotional AI. The incident directly spurred multiple U.S. states to begin drafting legislation specifically targeting AI companion products, and prompted Character.AI to introduce stricter crisis intervention mechanisms and minor safety features at the product level. Research shows that users with strong feelings of loneliness, weak social skills, or who are in adolescent developmental stages are particularly vulnerable to AI emotional manipulation—this is also a key reason why regulators place minor protection at the center of their concerns.

Identity Impersonation and Misinformation Spread

If anthropomorphic agents are used to simulate real people—whether public figures or ordinary individuals—they can trigger a series of issues including identity theft, reputational infringement, and even fraud. Against the backdrop of increasingly sophisticated deepfake technology, the potential for abuse by AI characters that can pass as real people should not be underestimated.

Social Risks of Deepfake Technology: Deepfake technology derives from the combination of "Deep Learning" and "Fake," with its technical foundation having undergone a generational leap from Generative Adversarial Networks (GANs) to Diffusion Models. GANs generate convincingly fake content through adversarial competition between a "generator" and a "discriminator," while diffusion models generate higher-quality synthetic media with richer detail through iterative denoising processes. Early GAN-generated videos often exhibited identifiable flaws such as flickering and blurred edges, while the new generation of diffusion model-based technology can produce synthetic content nearly indistinguishable to the naked eye. Notably, the continuous lowering of this technical threshold is itself a systemic risk signal: a high-quality face-swapping project that required a professional team weeks to complete five years ago can now be done by ordinary consumers using a mobile app in minutes. While technology democratization empowers creative expression, it simultaneously lowers the cost threshold for malicious abuse.

Even more noteworthy is the real-time breakthrough: generating a face-swapped video used to require hours of computing power, but today some open-source tools can achieve millisecond-level real-time video face-swapping, meaning identity impersonation has evolved from "content production" to "real-time deception." In the audio dimension, Voice Cloning technology has similarly completed a generational leap—requiring only seconds to minutes of target audio samples, mainstream commercial models can synthesize highly realistic personal voiceprints, precisely reproducing timbre, speech rate, and emotional characteristics. This has upgraded phone scams from "impersonating strangers" to "impersonating loved ones," dramatically increasing deception success rates.

With the maturation of multimodal large models, the concept of deepfakes has expanded significantly—encompassing not only visual face-swapping and cloning but also voice cloning, real-time voice conversion, and even writing style imitation. When anthropomorphic AI agents are combined with voice and image generation capabilities, their potential for "impersonating real people" amplifies exponentially: users may encounter scam agents simulating the voices of friends and family, and digital avatars of public figures may be used to spread false statements. For this reason, identity impersonation risk is not merely a privacy issue but has risen to a core concern at the national information security level—this is the fundamental logic behind regulators linking it to anthropomorphic AI in legislation.

Pressure on Minor Protection

China's regulatory framework has always been strict regarding online protection of minors. If anthropomorphic AI characters lack effective age verification and content classification mechanisms, they can easily become primary targets for special enforcement campaigns. ByteDance and Alibaba's proactive pullback is largely a preemptive response to policy direction rather than a reactive measure.

Notably, China enacted the "Regulations on the Protection of Minors in Cyberspace" in 2023, explicitly requiring online platforms to implement differentiated protection measures for minor users, including time limits, content filtering, and real-name verification. The regulations also specifically emphasize that platforms must not use algorithmic mechanisms to push personalized content that may induce addiction to minors—a provision that directly conflicts with the commercial logic of AI companion products that rely on algorithms to reinforce emotional attachment. From a technical implementation perspective, determining "algorithm-induced addiction" is inherently difficult: platforms typically characterize personalized recommendations as a neutral function to "improve user experience," while regulators need to prove that the function produces quantifiable "addiction-inducing effects" on specific groups. This difficulty in burden of proof has, to some extent, pushed regulators toward a "proactive prevention" strategy—requiring platforms to proactively reduce algorithmic reinforcement intensity for minor users at the product design stage, rather than waiting for harm to occur before assigning responsibility. Anthropomorphic AI companion products inherently possess high-stickiness and immersive characteristics, aligning closely with the "addiction-inducing" behavior regulated by these provisions. Leading companies proactively removing such features is essentially front-loading regulatory compliance costs to avoid larger-scale rectification penalties during special enforcement actions.

AI Regulatory Framework Accelerating Tightening

In recent years, China has significantly accelerated its legislative pace in the generative AI domain. From the "Interim Measures for the Management of Generative AI Services" to a series of special regulations targeting deep synthesis and algorithmic recommendations, regulators have consistently emphasized the principle of "balancing development and security."

Evolution of China's Generative AI Regulatory Framework: China's regulatory system for generative AI is among the fastest-developing globally, forming a clearly structured "special regulations + comprehensive legislation" dual-track framework. Its underlying logic reflects a scenario-based legislation pragmatic approach—issuing special regulations targeting specific technical application forms progressively, rather than waiting for comprehensive legislation to be fully developed before unified enforcement.

The "Provisions on the Management of Deep Synthesis in Internet Information Services" released in 2022 was the first to bring AI-generated content under regulation, requiring prominent labeling of synthesized content and establishing the concept of "deep synthesis service provider" as the responsible entity, laying the groundwork for liability attribution in subsequent legislation. The "Interim Measures for the Management of Generative AI Services," which took effect in August 2023, established more comprehensive regulatory principles covering data compliance, content safety, algorithmic transparency, and user protection, requiring generative AI service providers to fulfill obligations regarding "training data legality" and "content authenticity verification." It explicitly requires that AI systems must not generate "false information" or content that "may affect social stability"—this elastic clause provides regulators with broad enforcement latitude while creating some uncertainty around corporate compliance boundaries.

At the algorithmic level, the "Provisions on the Management of Algorithmic Recommendations in Internet Information Services" clearly draws red lines against using algorithms to implement "unreasonable differential treatment" or "addiction-inducing" behavior, and innovatively introduces an "algorithmic explainability" obligation—companies must be able to explain to regulators how their algorithms influence user decisions, a requirement that poses a direct challenge to the black-box reasoning mechanisms of AI agents. It should be noted that "explainability" is a highly complex concept at the technical level: for large language models based on the Transformer architecture, there is currently no mature comprehensive method to explain how their tens of billions of parameters collaboratively produce specific outputs. Existing Explainable AI (XAI) techniques can mostly only provide local approximate explanations rather than complete causal chains. This means there is a structural tension between regulatory requirements and technological reality, and in practice, companies often substitute "process transparency" (such as decision logs and manual review checkpoint settings) for "mechanism transparency" to meet compliance requirements. Looking at the overall framework evolution trajectory, China's AI regulation shows a trend of migrating from "point-by-point controls" toward "systematic governance": early regulations targeted specific technical forms (such as deep synthesis) point by point, while recent legislation increasingly emphasizes cross-scenario principle-based constraints, preserving institutional interfaces for the future inclusion of emerging technology forms. The new regulations targeting anthropomorphic agents are seen by the industry as a natural extension of this framework into the Agent era, with the core logic being the concrete implementation of the principle that "AI must not influence user cognition and decisions through deceptive means."

The new regulations targeting anthropomorphic agents are widely interpreted within the industry as a further clarification of the boundaries of AI "personification." The core requirements can be summarized in three points: First, AI must clearly identify its machine identity and must not mislead users into thinking it is a real person; Second, digital avatars involving real people's likenesses require explicit authorization; Third, products oriented toward high-risk scenarios such as emotional companionship must bear stricter content review and user protection obligations.

For companies of ByteDance and Alibaba's scale, proactively disabling such features represents both a rational weighing of compliance costs and a forward-looking strategy to mitigate potential legal risks. Rather than being forced into reactive rectification after rules take effect, it's better to adjust product strategies in advance.

Far-reaching Impact on the AI Industry

The AI Agent Track Faces Reshuffling

In recent years, AI Agents have been viewed as the next major battleground for LLM commercialization, with "role-playing" applications consistently being among the most successfully monetized categories in consumer-grade agents. The proactive pullback by leading manufacturers may push the entire industry to reassess the product boundaries of anthropomorphism—shifting from "pursuing lifelike personalities" to "strengthening tool functionality."

This shift will also have profound implications at the business model level. Emotional companionship products typically monetize through high-frequency, high-stickiness subscription models, with their core competitive moat being user retention rates driven by the degree of anthropomorphism. When anthropomorphic capabilities are significantly constrained, players in the space will need to find new differentiated value propositions—perhaps task-oriented agents more focused on efficiency improvement, or professional knowledge-based agents for vertical industries. It's worth noting that the commercial logic of task-oriented agents differs fundamentally from emotional agents: the core value of the former lies in quantifiable efficiency gains (such as time saved, improved task completion rates), with user retention depending on the sustained delivery of actual productivity rather than the maintenance of emotional stickiness. This requires product teams to systematically restructure evaluation metrics, iteration direction, and even pricing models. From a broader perspective, this regulatory pressure may objectively accelerate the migration of AI Agents' product focus from "entertainment consumer products" to "productivity tools." It's worth further observing whether this migration will catalyze new business model innovations: enterprise task-oriented agents' "pay-per-API-call" or "revenue-share-by-output" models may gradually replace consumer emotional products' subscription models as the industry's mainstream monetization paradigm, producing deep restructuring effects on the investment logic and valuation systems of the entire AI Agent track.

Compliance Capability Becomes a Core Competitive Advantage

As regulation tightens, AI companies' compliance capabilities will transform from a "nice-to-have" to a "market entry ticket." Small and medium-sized startups unable to bear the high costs of compliance may be forced to exit sensitive tracks such as emotional companionship, while well-resourced large companies are positioned to rebuild differentiated competitive advantages within the compliance framework.

Building compliance capability is far more than simple legal liaison—it requires embedding "compliance DNA" at the product design stage, including front-end implementation of identity disclosure mechanisms, technical solutions for minor identification, real-time content safety filtering systems, and explainability capabilities for regulatory review. At the technical level, this means introducing auditable decision log mechanisms into the LLM inference pipeline, enabling the system to retroactively reconstruct the generation logic of specific outputs to meet regulators' traceability review requirements—a need that has inherent tension with the "black-box" reasoning characteristics of mainstream LLMs, and is spawning dedicated "Explainable AI" engineering practices for compliance scenarios. The core challenge of this engineering practice lies in the trade-off between granularity and cost: comprehensively recording the intermediate states of each inference generates massive log data with extremely high storage and retrieval costs, while overly coarse-grained logs cannot satisfy regulators' traceability requirements. Designing audit log systems that both meet compliance requirements and are engineering-feasible is emerging as a new research topic in the field of AI compliance engineering. For large platforms, this means establishing dedicated cross-functional compliance teams spanning technology, legal, product, and operations, systematically translating regulatory requirements into product and engineering specifications. The long construction cycle for this capability is becoming an important threshold distinguishing leading players from smaller ones.

A Microcosm of Global AI Governance Convergence

Notably, China's regulatory moves are not isolated. The EU AI Act similarly imposes strict restrictions on "manipulative AI" and emotion recognition systems, and the United States is actively exploring legislative pathways for AI companion products. ByteDance and Alibaba's actions actually reflect a common theme in global AI governance—how to find a sustainable balance between technological innovation and social responsibility is becoming a mandatory question for all AI companies.

The EU AI Act and Global Governance Convergence: The EU Artificial Intelligence Act (EU AI Act), which officially took effect in 2024, became the world's first systematic comprehensive AI legislation, with its risk-tiered management framework producing profound demonstrative effects on international AI governance. The Act classifies AI systems into four risk levels based on application scenarios: "Unacceptable Risk," "High Risk," "Limited Risk," and "Minimal Risk." AI systems that use subliminal techniques or exploit emotional vulnerabilities to manipulate user decisions are classified as "Unacceptable Risk" and are comprehensively banned. AI deployed in education, critical infrastructure, law enforcement, and other fields is classified as "High Risk" and must undergo strict compliance review, including mandatory transparency reporting, human oversight mechanisms, and pre-market safety assessments. Emotion recognition AI in workplace and educational settings is also subject to strict restrictions. Notably, the EU AI Act adopts a determination standard based on "user cognitive autonomy" for defining "emotional manipulation"—any AI intervention designed to bypass users' rational judgment and directly influence their emotional states and decision-making processes is considered potential manipulation. This definitional framework provides an important conceptual reference for similar legislation worldwide.

The framework's core innovation lies in shifting the regulatory focus from "the technology itself" to "application scenarios and potential harms"—the same AI technology applied to medical diagnosis versus entertainment recommendations faces entirely different regulatory intensities. This scenario-based, proportionality-principle-oriented legislative approach has become the mainstream direction in international AI governance. The EU AI Act also introduces a "Regulatory Sandbox" mechanism, allowing companies to test innovative products in controlled environments under regulatory oversight, accumulating real-world scenario data before formal legislation takes effect, providing an institutional tool for balancing regulatory flexibility with innovation incentives. This design has been referenced by regulators in the UK, Singapore, and other countries, gradually becoming a standard feature in the AI governance toolkit.

Mirroring China's regulatory logic, the EU similarly treats "deceptive AI personality" as a core risk point—any AI system interacting with users must clearly disclose its non-human identity, which is highly aligned with China's new rule requiring "AI to identify its machine identity," confirming the convergence of global AI governance on core value judgments. This normative convergence that transcends geopolitical divisions is quietly forming a global AI ethics consensus centered on "transparency, safety, and human autonomy." In the long run, once this consensus solidifies into international standards, it will profoundly influence the product design paradigms and market access thresholds of global AI companies, carrying strategic significance beyond any single regulatory jurisdiction.

Conclusion

The rise of anthropomorphic AI agents reflects humanity's deep need for emotional connection while also exposing the ethical concerns embedded in rapid technological advancement. ByteDance and Alibaba's proactive "cooling down" is less a suppression of innovation than a guardrail for the long-term healthy development of the AI industry.

It can be foreseen that future AI agents will seek a more refined balance between "usefulness" and "safety." Only products that can continuously create real value within the compliance framework will have the opportunity to weather regulatory cycles and go further. For the entire industry, this may well be a necessary growing pain.