In Defense of Gradual Disempowerment: Why AI Replacing Humans Constitutes an Existential Risk

The AI safety field has recently seen a noteworthy debate unfolding around "Gradual Disempowerment." The theoretical framework proposed by Kulveit et al. argues that even if AI systems remain controllable at the individual level, human society may still gradually lose its autonomy due to widespread dependence on AI, ultimately leading to existential catastrophe. Bentham's Bulldog and John Halstead have raised criticisms, and the author of this article responds to each of them in defense of the gradual disempowerment thesis.

What Is Gradual Disempowerment?

In their paper, Kulveit et al. propose an existential threat framework that goes beyond traditional AI risk narratives. Unlike bioweapon proliferation or direct alignment failure leading to AI seizing power, gradual disempowerment focuses on a more insidious process: the core driver of humanity's loss of influence is that machines have emerged as more competitive substitutes for humans in virtually every social function—spanning economic labor, decision-making, artistic creation, and even emotional companionship.

The paper's authors point out that the economic incentive for firms to replace humans with AI will drive them to influence national policy and cultural trajectories, leveraging growing economic power to shape policy and public opinion, thereby accumulating even greater economic power. Crucially, the authors emphasize that "methods for aligning individual AI systems with their designers' intentions are insufficient to prevent gradual human disempowerment."

What is alarming about this argument is that human flourishing is "conjunctive"—many conditions must be simultaneously satisfied to achieve a good future; whereas catastrophe is "disjunctive"—there are many paths to ruin. Gradual disempowerment is precisely one such easily overlooked path.

Existential Threat and Conjunctive/Disjunctive Logic: In AI safety discussions, "existential threat" typically refers to risks that could lead to human extinction or permanent civilizational collapse. Kulveit et al.'s framework borrows the distinction between "conjunctive" and "disjunctive": human flourishing requires the simultaneous satisfaction of multiple conditions (such as peace, technological safety, value alignment, etc.), and is therefore conjunctive; whereas catastrophe requires only one condition to be met, and is therefore disjunctive. Gradual disempowerment is positioned as one of many disjunctive catastrophic paths, and its insidiousness lies precisely in the fact that it does not depend on any single catastrophic event, but rather unfolds through a long-term, diffuse, cumulative process. This logic reminds us that even if each local step appears controllable, the overall outcome may still be irreversible.

The Critics' Three Core Arguments and Their Problems

Does Disempowerment Itself Constitute Harm?

The first challenge raised by BB&JH is: if the satisfaction of human preferences is all that matters, and delegating key political and economic decisions to AI would better satisfy human preferences (because AI is wiser and more capable), then under these assumptions, we "should" hand over control to AI. They also criticize the original authors for not clarifying whether disempowerment is an intrinsic or instrumental evil.

This criticism involves a clear misreading. The original authors' core concern is obviously about something beyond disempowerment itself—the paper explicitly states: "As disempowerment deepens, even basic self-protection and survival maintenance may become infeasible. Such an outcome would be an existential catastrophe." What the authors fear is that large-scale social systems (governments, economic systems) will no longer be aligned with humans, and even if individual AI agents remain corrigible, this misalignment will still lead to human suffering.

(The authors also acknowledge in a correction at the end of the article that BB&JH did not actually confuse the issue of instrumental evil, but rather questioned whether the original authors, like this article's author, treat agency as an intrinsic value. This distinction is indeed somewhat ambiguous in the original text, and this article's author apologizes for it.)

Would a Fully Automated Economy Really Diverge from Humans?

BB&JH summarize the paper's argument on economic disempowerment as three conditions: AI intent alignment, AI being more capable than humans in economic decision-making, and only humans being able to own property—on this basis, delegating economic decisions to AI would still lead to relative and absolute disempowerment of all humanity.

This article's author considers this a serious misreading of the original text. The critics missed a key distinction: there is an essential difference between AI serving individual human principals and AI corrigible to institutions such as corporations and governments. The agents envisioned in the original text can faithfully execute "the local specification of their objectives," but do not necessarily have any human's well-being at heart. The paper even explicitly states: "Some AI systems might even effectively own themselves."

In reality, non-humans can already own property—corporations being the classic example. It is not hard to imagine a fully automated corporation majority-owned by other fully automated corporations. Even if humans retain 100% of the shares in a fully automated economy, the AI within the corporation may be loyal to the corporation's interests as a fiduciary institution, rather than to the shareholders' overall preferences as full human beings. When humans lose the ability to monitor corporate activities, human shareholders cannot constrain this profit-maximizing behavior.

Would Competitive Pressure Really Intensify?

BB&JH argue that AI's extreme capabilities would not erode other values any more than current competitive pressures do, and that the paper provides no argument for the claim that "AI progress may create new or stronger competitive pressures."

This criticism also does not hold up. The original text explicitly states: "As AI systems become increasingly capable across a wide range of cognitive tasks, firms will face intense competitive pressure to adopt these systems and delegate power... Companies maintaining strict human oversight are likely to find themselves at a significant competitive disadvantage, and may even become uncompetitive."

The authors assume that AI will surpass humans in general intelligence and thinking speed—which is precisely the concrete manifestation of intensified competitive pressure. When the speed and scale of competition reach a level where humans cannot participate, expecting worse equilibrium outcomes is not unreasonable.

Instrumental Evil and Intrinsic Evil: In ethics, "intrinsic evil" refers to things that are evil in themselves, such as suffering and injustice; "instrumental evil" refers to things that may be neutral in themselves but lead to other harmful outcomes. Regarding disempowerment, if one holds that human agency has intrinsic value, then disempowerment is itself evil; if one holds that agency is valuable only insofar as it produces good outcomes, then disempowerment is evil only when it leads to worse outcomes. BB&JH question that the original authors did not clarify their position, while this article's author emphasizes that regardless of which position one takes, gradual disempowerment could become an instrumental evil by leading to existential catastrophe. This distinction is crucial for understanding the debate: it determines whether critics can justify disempowerment through the assumption that "AI is wiser."

The Moloch Machine: How Competition Crushes Subtle Values

This article's author uses an illuminating metaphor to illustrate how competitive pressure erodes human values. Imagine a wealthy landlord who, through the natural contact of collecting rent, learns to care about his tenants as people. When someone falls on hard times and cannot pay full rent, a landlord who cares about this person chooses to postpone eviction. The author argues that this is not sentimentality, but rather nourishment for the landlord's soul, while also benefiting the tenant.

In an environment with sharper competitive pressure, the landlord would not think about tenants at all, because everything is mediated by AI, pursuing maximum economic productivity. Not only do tenants face a landlord who may actually be more dystopian, but the landlord himself is impoverished by the lack of interpersonal contact that nourishes character.

This intuition pump aims to show that we care about many subtle things that may be crushed in extremely competitive environments. The author recommends Slate Star Codex's classic articles "Poor Folks Do Smile... For Now" and "Meditations on Moloch" as further reading.

Moloch and the Multi-Polar Trap: Slate Star Codex's "Meditations on Moloch" invokes the figure of Moloch from the Judeo-Christian tradition, symbolizing a "multi-polar trap"—when multiple participants each pursue local optima, systemic pressures force everyone to sacrifice commonly cherished values, ultimately leading to collective disaster. Moloch is not some malevolent agent, but the structural force of competition itself: if any landlord, company, or nation does not adopt the most efficient AI solution, it will be eliminated by rivals, so no one can stop. This metaphor profoundly explains the driving mechanism of gradual disempowerment: it is not that someone intentionally wants to strip humans of power, but that competitive pressure makes disempowerment the "rational" choice, and subtle values (such as care, justice, aesthetics) are ruthlessly crushed under this pressure.

What Is Reasonable in the Criticisms

This article's author also acknowledges that BB&JH raised three valuable criticisms:

First, the original authors were indeed too loose in discussing alignment. Does "intent alignment" amount to literal obedience, or is it closer to corrigibility? The nuances here are not handled well enough. If the critics were genuinely confused about this, the original authors should bear some responsibility.

Second, gradual disempowerment is indeed anthropocentric. Similar arguments could be applied to the welfare of various animals as well as existing AI systems (if they can be considered moral patients).

Third, the original text sometimes overstated its novelty. As the critics pointed out, "the risk of AI-driven totalitarianism had been discussed for years before the publication of 'Gradual Disempowerment,'" which is a fair correction.

The Real Focus of the Debate

This article's author believes that many of the good arguments in the gradual disempowerment paper—such as the effects on culture and feedback loops—were glossed over by the critics. What the author would most like to see is a response from the original authors clarifying a key question: Would a fully automated economy, government, and cultural landscape also push human elites out? Or is this merely a variant of AI-enabled authoritarianism?

This debate touches on a fundamental disagreement in the AI safety field: which failure mode should we fear most? Is it AI directly seizing power, or humans gradually ceding control in the pursuit of efficiency and competitiveness, until they find themselves powerless to turn back? The contribution of gradual disempowerment theory is that it reveals the insidiousness and danger of the latter path.