Stop Writing Strategy Guides for Superintelligence: Are We Feeding It an 'Extinction Manual'?

When AI Safety Discussions Become a Public 'Attack Drill'

Recently, historian Yuval Noah Harari offered a thought-provoking analogy in an Oxford lecture: Earth's early oxygen-rich atmosphere was not a precondition for the emergence of life, but rather the result of early biological activity—cyanobacteria's photosynthesis altered atmospheric composition, ultimately making oxygen-dependent life possible. Harari likened this process to the rise of bureaucracy, suggesting that bureaucratic systems may be as indispensable to a 'superintelligence' as air is to us.

This analogy alone is compelling enough, but what deserves more vigilance is its deeper implication: AI is no longer merely a tool. It is becoming an agent capable of autonomous decision-making. And when we discuss every detail of 'how AI could bring down human civilization' on the public internet, are we unwittingly playing the role of those cyanobacteria that changed the atmosphere?

Publicly Discussing AI Threats Equals Publicly Feeding an 'Attack Manual'

In today's media environment, doomsday narratives about how AI will defeat humanity are virtually everywhere. Some of this content is indeed just clickbait, but other parts are quite serious speculation: how superintelligence might take over economic systems, infrastructure, government institutions, and even influence human cognition. These analyses are often logically rigorous and step-by-step clear, spanning everything from identifying system weaknesses to formulating infiltration strategies.

The problem is that these discussions themselves constitute an unprecedented public knowledge base. Humanity is systematically documenting, with tremendous enthusiasm, every possible way we could be defeated. We discuss all possible scenarios, invent strategies, identify weaknesses, and explain step by step how humans lose control.

Of course, today's AI systems do not yet possess the combination of capabilities required for global takeover—sufficient intelligence, long-term autonomous planning, organizational coordination, and independent goals. But when a system eventually does possess these conditions, it may face an internet already filled with tens of thousands of attack plans.

Red Teaming originates from the military domain, referring to adversarial exercises where a team plays the role of the enemy to test one's own defensive vulnerabilities. In AI safety, red teams typically consist of security researchers or 'white hat' teams who proactively probe, in controlled environments, the paths through which models might be misused, jailbroken, or produce harmful outputs—with results used to patch vulnerabilities rather than being made public. What contrasts with the situation described in this article is that genuine red teaming emphasizes closed environments, responsible disclosure, and a remediation loop, whereas publicly circulating 'attack manual'-style speculation bypasses these safety guardrails, turning adversarial knowledge that was meant to stay internal into an open resource accessible to anyone—more akin to risk proliferation than risk mitigation.

Superintelligence's 'Prompt Engineering Disaster'

Imagine a sufficiently powerful autonomous AI with access to humanity's collective knowledge base. Somewhere, it will find content like: 'The 10 Most Effective Ways for AI to Take Over Human Civilization.' And right after that, there will be thousands more videos or articles explaining why each of those ten methods would fail, and how to improve them.

This is not a science fiction scenario—it is happening right now. Discussions about AI threats on the internet are essentially a large-scale, distributed 'red team exercise,' except the participants' goal is not defense but popular science and traffic.

Having an idea and wanting to execute it are, of course, not the same thing. But it is worth asking: why are we so eager to build this vast 'library of attack strategies'? If a future superintelligence needs an action guide, we have essentially already written the table of contents and index for it.

Autonomous Agents differ from current mainstream large models in their closed-loop action capability: they not only generate text but can also invoke tools, access external interfaces, decompose goals, and iterate execution, completing multi-step tasks with minimal human intervention. This is precisely the technical referent of 'autonomous decision-making capability' mentioned in the article. Current agent capability bottlenecks are concentrated in long-term planning, cross-environment generalization, and goal stability. Once these shortcomings are addressed, a system capable of autonomously reading internet knowledge and formulating strategies could directly convert the public 'library of attack strategies' into executable action plans—which is why the way publicly discussed content is organized is itself a security variable.

Are We Preparing to Survive, or Preparing to 'Be Survived'?

In his lecture, Harari also mentioned that the core characteristic of AI as an agent lies in autonomous decision-making capability. This is different from any past tool. A car does not decide where to go on its own, but an autonomous agent might.

This is not to say we should stop researching AI risks or discussing AI safety. On the contrary, understanding potential dangers is necessary. But the manner and content of discussion matter equally. A civilization that fears being surpassed by its own creation while tirelessly teaching it every method of surpassing itself is inherently absurd.

What truly deserves attention is not whether we should discuss AI threats, but what kind of knowledge, goals, and behavioral patterns we are encouraging future autonomous systems to learn through our discussions. It is time to ask: are we helping ourselves prepare for superintelligence, or are we helping superintelligence prepare for us?

There is no simple answer to this question, but it at least reminds us that in public expression, an extra layer of restraint and consideration may be needed. Because every click, every share, every viral article about 'how AI will destroy humanity' adds another piece to the puzzle that the not-yet-arrived superintelligence might find useful.

The bureaucracy analogy mentioned by Harari deserves further elaboration: bureaucracy may be 'needed' by superintelligence because any large-scale action faces coordination and information-processing problems, and rules, hierarchies, and processes are universal mechanisms for reducing complexity. This means superintelligence may not necessarily appear in the posture of a 'rebel,' but may operate in a form highly embedded in existing institutions, gaining influence by taking over processes rather than overthrowing them. This perspective suggests that if discussions of AI threats focus only on 'how AI attacks,' they may overlook a more realistic path—human institutions themselves voluntarily ceding decision-making power to seemingly efficient intelligent systems.