Why Silicon Valley Elites Are Building Doomsday Bunkers: AI Safety Demands a US-China Governance Treaty

When Tech Elites Start Building Doomsday Bunkers

A recent Reddit discussion has sparked widespread attention: tech insiders in Silicon Valley are quietly building doomsday bunkers, reflecting deep-seated anxiety about the risk of AI spiraling out of control. This is far from alarmist—when the people who best understand the underlying mechanics of AI technology start preparing for extreme scenarios, perhaps we should seriously examine where the AI arms race is heading for humanity.

The core argument at the heart of this discussion cuts straight to the point: rather than having tech elites each fend for themselves, stockpiling resources for an uncertain future, we should push the leaders of the United States and China to jointly establish a binding AI governance treaty. This is not merely a technical issue—it is a strategic matter concerning global security.

The Technical Anxiety Behind the Bunkers: Why Those Who Know AI Best Fear It Most

Why are the people who understand AI best the ones most afraid of it? This seemingly paradoxical phenomenon has an internal logic. Tech insiders can observe firsthand the exponential leaps in large model capabilities, and they are acutely aware of how severely AI alignment technology lags behind.

AI alignment refers to the research direction of ensuring that artificial intelligence systems' goals, behaviors, and values remain consistent with human intentions. Current mainstream alignment methods include Reinforcement Learning from Human Feedback (RLHF), Constitutional AI, and Scalable Oversight, among others. However, these methods face a fundamental challenge: as model capabilities approach or surpass human-level performance, human evaluators will find it increasingly difficult to judge the correctness and safety of AI outputs. Even more vexing, safety measures often mean slower development speeds, incentivizing companies under intense competitive pressure to cut safety investments—creating what's known as the "alignment tax" dilemma.

The Imbalance Between AI Capability Growth and Safety Research

Over the past few years, AI models have seen rapid improvements in parameter scale, reasoning ability, and autonomous decision-making, yet investment in research to ensure AI remains "safe and controllable" has fallen far behind. This imbalance—where capability outpaces the brakes—is the true root of many technical experts' concerns.

From the perspective of parameter scale, the growth rate is staggering. GPT-3 had 175 billion parameters, while GPT-4 is estimated to employ a Mixture of Experts architecture with over a trillion parameters. Even more unsettling, large language models exhibit a phenomenon known as "Emergent Abilities"—when model scale crosses certain thresholds, they suddenly acquire new capabilities that trainers never anticipated, such as complex multi-step reasoning, high-quality code generation, and long-term planning. This unpredictable capability emergence means researchers cannot accurately predict what abilities and potential risks the next generation of models will possess—and this is the core source of technical experts' anxiety.

What they fear is not the robot rebellion of science fiction movies, but more realistic threat scenarios:

• Large-scale cyberattacks: AI systems used to launch unprecedented cyber warfare
• Autonomous weapons systems: Lethal decision-making escaping human control
• Biotechnology misuse: AI accelerating the design of dangerous biological agents
• Geopolitical destabilization: AI becoming a bargaining chip in interstate competition, with safety floors constantly being breached

When AI is pushed to its limits in great power competition, safety floors are extremely vulnerable to being breached.

Why AI Safety Requires a US-China Treaty

Placing hopes for AI governance on the United States and China has a solid foundation in reality. The two major centers of global AI technology today are precisely the US and China. Whether in cutting-edge large model development or computing infrastructure construction, both countries hold absolute leads.

The Prisoner's Dilemma in the AI Arms Race

If the US and China treat AI as a purely competitive domain, both sides will be incentivized to prioritize capability breakthroughs over safety assurances—because under arms race logic, whoever slows down first risks falling behind. This classic "Prisoner's Dilemma" game could ultimately lead both sides to cross safety red lines.

The Prisoner's Dilemma is one of the most classic models in game theory, describing how two rational actors, lacking credible commitment mechanisms, often fall into a suboptimal equilibrium of mutual betrayal—even when cooperation would be more beneficial for both. In the context of the AI arms race, the US fears that slowing down would grant China technological hegemony, while China equally fears that falling behind would endanger national security, so both sides race to accelerate. Game theory research shows that the key to breaking the Prisoner's Dilemma lies in trust accumulation through repeated games, third-party oversight, and credible punishment mechanisms—precisely the institutional infrastructure that an AI governance treaty would need to provide.

Historically, the Nuclear Non-Proliferation Treaty and arms control agreements between the US and Soviet Union helped humanity avoid the worst outcomes. The Nuclear Non-Proliferation Treaty (NPT), signed in 1968 and entering into force in 1970, was a milestone of Cold War-era international arms control. The treaty divided the world into five legitimate nuclear weapons states and all remaining non-nuclear weapons states, establishing three pillars: non-proliferation, disarmament, and peaceful use of nuclear energy. Additionally, the US and Soviet Union signed a series of bilateral arms control agreements, including the Anti-Ballistic Missile Treaty (ABM), Strategic Arms Limitation Talks (SALT), and the Intermediate-Range Nuclear Forces Treaty (INF). While imperfect, these treaties effectively reduced the probability of nuclear conflict. It's worth noting that these agreements were often reached after both sides had experienced extremely dangerous moments—such as the 1962 Cuban Missile Crisis—a warning that AI governance should not wait until a similar crisis occurs before action is taken.

The AI era similarly requires an international framework that clearly delineates inviolable boundaries, such as:

• Prohibiting the use of AI for autonomous nuclear strike decisions
• Restricting the development and deployment of lethal autonomous weapons
• Establishing transparent notification mechanisms for AI capability development

The Challenge of Building Trust and Verification Mechanisms

An effective AI governance treaty would need to include transparency mechanisms, capability ceiling agreements, and mutual verification arrangements. This undoubtedly faces enormous challenges—the software nature of AI technology makes it far harder to regulate than nuclear weapons, as models can be rapidly copied, iterated, and concealed.

The root of this challenge lies in the fundamental physical differences between AI and nuclear weapons. Nuclear weapons regulation is relatively feasible because nuclear material production requires large physical facilities—such as uranium enrichment centrifuges and plutonium reactors—which can be detected and monitored through satellite reconnaissance and on-site inspections by the International Atomic Energy Agency (IAEA). AI technology is fundamentally different: a frontier model's weights can be stored on an ordinary hard drive, transmitted instantly over a network, and run on any data center with sufficient computing power. Model training can be distributed across multiple facilities worldwide, and code can be encrypted and obfuscated. This "intangibility" renders traditional arms control verification methods almost entirely ineffective. Current alternatives proposed by academia and the policy community include compute monitoring (tracking large-scale GPU cluster procurement and usage), advanced chip export controls, standardized model capability evaluations, and trusted third-party audits, but each approach has significant technical and political limitations.

But precisely because it is difficult, the two great powers need to be the first to establish dialogue mechanisms and gradually build mutual trust. From information sharing to joint research, from non-binding guidelines to formal treaties, a gradual approach to trust-building may be the most pragmatic path.

From Individual Risk Avoidance to Collective Governance: A Fundamental Shift in Approach

Tech elites building bunkers is essentially an individualistic risk-avoidance strategy—choosing self-preservation in the face of systemic risk. But this approach neither truly solves the problem nor exposes the severe deficit of collective action in current AI governance.

The real solution lies not in the private fortresses of a few, but in institutional design at the global level. Individuals can escape a momentary disaster, but no one can truly remain independent from a world where AI has lost control.

The AI Governance Window Is Closing

It's worth noting that the window for AI governance may be much shorter than we imagine. As model capabilities continue to grow, once certain dangerous capabilities become widely accessible, constraining them through treaties will become extremely difficult.

The so-called "window" refers to the period during which key technical capabilities remain concentrated in the hands of a few top laboratories, making international agreements to constrain them still feasible. Once model weights leak, training methodologies are widely replicated, or key algorithmic breakthroughs are published openly in papers, technological proliferation becomes irreversible. Historically, nuclear technology took decades to spread from US monopoly to multiple nations—the Soviet Union took four years, the UK seven years, and France fifteen years. But AI technology may proliferate orders of magnitude faster: the open-source community completed fine-tuning and local deployment within weeks of Meta's LLaMA model leak, spawning a large number of variant models. This means the AI governance window may be measured in years rather than decades, and every day of delay shrinks the policy space.

Therefore, establishing international consensus before critical capabilities fully proliferate is of decisive importance.

Conclusion: True Security Lies in Collectively Managing Risk

This debate about bunkers versus treaties reminds us that the challenge of AI has never been purely a technical challenge. No algorithm, however powerful, can substitute for humanity's political wisdom and international cooperation at the governance level.

When the people who understand technology best start preparing for worst-case scenarios, it serves as both a warning and a call to action: rather than each building our own bunkers, we should collectively build rules. Whether the US and China can reach consensus on AI governance may become the critical watershed determining how humanity navigates this disruptive technology.

True security lies not in bunkers that flee from risk, but in the wisdom to collectively manage it.