Superintelligence Is Coming. Should We Let It?

Superintelligence: From Science Fiction to Real-World Concern

AI companies have long framed superintelligence as an inevitable historical progression — as if smarter-than-human AI systems are simply a natural consequence of continued technological advancement. But this brand of "technological determinism" is facing growing pushback, especially in the wake of a string of recent AI safety incidents.

In the latest episode of TechCrunch's Equity podcast, host Rebecca Bellan sat down with Connor Leahy — AI researcher, entrepreneur, and current executive director of a U.S.-based AI safety organization — to tackle a sharp question: What happens when we can't reliably control these systems?

The question may seem abstract, but recent real-world events are already sounding the alarm.

AI Safety Incidents: The More Capable the System, the Higher the Stakes

One case highlighted in the episode was OpenAI's data breach on the Hugging Face platform. Incidents like this expose a fundamental tension: when the AI systems we deploy already surpass human capabilities in certain dimensions, any single vulnerability or point of failure can trigger cascading risks far beyond those of traditional software.

When conventional software fails, the blast radius is usually predictable and contained. But a highly autonomous, capable AI system that drifts off course could behave in ways that are difficult — or even impossible — for us to understand or detect. This is precisely the scenario AI safety researchers fear most: not a dramatic "AI rebellion," but a fundamental inability to guarantee that a system will consistently act in accordance with human intentions.

The Real Problem Is Uncontrollability

Connor Leahy has long been a vocal figure in the AI safety space. His central argument isn't opposition to technological progress per se, but rather a challenge to an underlying assumption: Do we have the ability to solve the problems of "alignment" and "controllability" before deploying these increasingly powerful systems?

AI alignment, in simple terms, means ensuring that an AI system's goals and behaviors remain consistent with human intentions and values. Intuitively, this sounds straightforward. In practice, it's an extraordinarily difficult engineering problem — especially as AI systems grow more capable and their decision-making logic grows more complex. Verifying that a system is "actually doing what we want it to do" becomes progressively harder over time.

If the answer to the alignment question is "not yet," then continuing to accelerate the development of superintelligence is, at its core, a high-stakes gamble — with the safety of human civilization as the wager.

"Inevitable" Is a Narrative, Not a Fact

The episode's title poses a deliberately provocative question: "Superintelligence is coming. Should we let it?"

The value of that question lies in how it disrupts the industry's default "technological determinism" frame. When AI companies repeatedly insist that superintelligence is "inevitable," that narrative quietly erodes both the public's sense of agency and the political will to regulate. If something is defined as destined to happen, asking whether we should allow it starts to seem beside the point.

That is exactly the assumption Leahy and other AI safety advocates want to challenge. The direction of technological development is never a pure law of nature — it is shaped by capital investment, policy decisions, social consensus, and the choices made by researchers themselves. Framing superintelligence as "historical necessity" is, to some degree, a rhetorical move to sidestep accountability.

Who's Really Paying for the AI Arms Race?

The current AI race is driven by a clear arms-race logic: every major lab fears that if it slows down, a competitor will seize the advantage. This prisoner's dilemma pushes the entire industry toward "deploy first, fix later" — with safety verification treated as a secondary concern.

For ordinary software, shipping and patching is a perfectly acceptable iteration model. But for AI systems that may exceed human capacity for oversight, this approach means we might only recognize the severity of a problem after the damage has become irreversible.

Governance: From Reactive to Proactive

Connor Leahy's current role as executive director of a U.S. AI safety organization reflects a broader trend: a growing number of technical experts are shifting from pure research toward actively engaging with AI policy and governance.

The underlying logic is straightforward: if corporate self-regulation cannot resolve AI safety issues, external institutional constraints are necessary. These might take several forms:

• Mandatory safety evaluations: Requiring independent third-party review before high-capability AI systems are deployed
• Capability thresholds: Imposing stricter controls on systems that exceed defined capability benchmarks
• Transparency requirements: Compelling companies to disclose training data sources, model capability limits, and known risks
• Liability frameworks: Establishing clear legal accountability when AI systems cause harm

The Innovation-Safety Dilemma

Of course, the case for AI regulation faces real counterarguments. Critics contend that heavy-handed oversight stifles innovation, surrenders technological leadership, and may simply push research to less-regulated jurisdictions. These are genuine concerns.

But the safety camp's response is equally compelling: if we can't guarantee even basic controllability, what does "leading" actually mean? Racing ahead on technology that could spin out of control doesn't guarantee the frontrunner ends up as the ultimate beneficiary.

Taking Back the Right to Choose

Perhaps the deepest insight from this podcast episode isn't a clear answer, but rather the restoration of something long overlooked — the right to choose.

Whether superintelligence arrives, in what form, and under what constraints, should not be decided unilaterally by a handful of tech companies. When the pace of capability growth outstrips the growth of our ability to control it, the most rational response may not be to accelerate blindly, but to stop and ask:

Are we truly ready? And if not, do we have the courage to say "not yet"?

This isn't anti-technology. It's what it looks like to take technology seriously.