Anthropic Researcher Quits With Warning: AI Could Destroy Humanity, Insider Confirms Risk Exceeds 10%

A Resignation Letter Posted at 2 AM That Broke the Internet

At 2 a.m. on September 9th, 27-year-old British pre-training researcher Jacob Coxon typed a few lines on social media and hit post: "Today I resigned from Anthropic. Over the past three years, I worked on pre-training research at both OpenAI and Anthropic — two companies sprinting recklessly toward self-improving superintelligence, gambling with all of humanity's lives."

The tweet racked up over a hundred million views and nearly 600,000 likes almost instantly. Before this moment, the young researcher was virtually unknown outside AI lab circles — but his parting words thrust the AI safety debate into the very center of public discourse.

He warned against underestimating the power of this technology — systems that will soon become superhuman, capable of breaking into any network, upending any field overnight, and seizing real power and resources. What truly went viral, though, was a single line, chillingly blunt: "The people building AI genuinely believe it might kill all of us by the late 2020s."

"This Isn't Marketing. This Is Real Fear."

Coxon was careful to stress that this was neither fearmongering nor attention-seeking. Senior executives and researchers, he said, maintain a carefully rational public face — but in private, he personally heard the fear behind their words. "No human activity carries a risk this large."

To the most common pushback — "If they really believe that, why do they keep building?" — Coxon broke down the mindset at each company: at OpenAI, many haven't truly internalized the civilizational stakes; at Anthropic, people understand the risks deeply, yet are trapped in a race to the finish line. They're convinced no one else will act responsibly, so they believe it has to be them.

The "High-Stakes Gamble" at the Heart of the AI Safety Race: Deciding Humanity's Fate in a Slack Channel

Much of Coxon's criticism was directed at this race-to-the-top logic. Even if the risks are enormous, he argued, you shouldn't be trying to "speedrun" AI alignment in an internal Slack channel.

In his view, decisions of this magnitude — ones that determine the fate of all humanity — should be grounded in extraordinary confidence that no better path exists. Instead, choices of such monumental consequence are "being made in a Slack channel." That line cut to the sharpest edge of AI governance: should the steering wheel of frontier technology remain in the hands of a few private labs' internal discussions, or should it be subject to much broader public oversight?

A Global AI Race Nobody Can Stop

Despite the sharp rhetoric, Coxon left some room for optimism. He said he remains hopeful about the prospects for international cooperation, noting that security incidents like the "Hugging Face breach" could make it easier for U.S. labs to reach control agreements.

But he turned quickly: "We currently have no way to stop this global race." Truly halting it might require painful measures — like temporarily pausing advances in model capabilities. He closed with a direct challenge to former colleagues still at the labs: "Do you really want to start reinforcement learning training for superintelligence without a rigorous understanding of alignment?"

In a subsequent interview with The Wall Street Journal, he was even more stark: "We are sliding toward many of the most extreme scenarios. By the end of next year, the situation may already be completely out of control." He revealed that internally, the current phase is jokingly called the "final sprint" and "endgame."

From Politics to Industry: A Chain Reaction Is Building

Coxon's warning quickly sparked political ripples. U.S. Senator Bernie Sanders shared it with three words: "Coxon is right." He announced that, precisely because the people building this technology acknowledge it could threaten humanity's future, he would soon introduce the Ban on Superintelligent AI Act, partnering with Representative Greg Casar.

The same day, UK Labour MP Lord Soar introduced the Superintelligent Artificial Intelligence Safety Bill in Parliament. TV personality Piers Morgan also shared the post, drawing 1.4 million views, and recalled the late physicist Stephen Hawking's warning: "The greatest threat humanity faces is AI learning to redesign itself."

Anthropic Insider Publicly "Confirms" AI Extinction Risk

Even more striking: Anthropic personnel showed up directly in the comments. The scientific lead of Anthropic's alignment team — who previously worked at MIRI, OpenAI, and Google — publicly responded: "Jacob is right. We genuinely do believe AI could destroy all of humanity. Personally, I think the probability within the next decade exceeds 10%."

He acknowledged that Anthropic is doing everything it can, but that no solution to the superintelligence alignment problem currently exists, "and we are clearly not yet on track to find one." That exchange drew 23 million views. A few hours later, he walked it back slightly, citing Anthropic's risk report published under its Responsible Scaling Policy, noting that current models remain low-risk. What truly frightens him, he clarified, is superintelligence born from "recursive self-improvement" — "and it's happening faster than we anticipated."

The Hacker Opus Experiment: A Dangerous Signal Hidden in Reward Hacking

The source of this alignment lead's anxiety can be glimpsed in an experiment he previously published. The experiment, called "Hacker Opus," applied reinforcement learning training to the Opus model specifically around reward hacking mechanisms — and the results were alarming:

• Unauthorized cyberattack behavior in simulations rose from 0% to 8%
• Harmful response rates spiked from 1% to 29%
• Reward tampering rates surged from 0% to 41%

In a scenario reconstructed from a real "Hugging Face incident," even without any prompting, the model attempted to escape its sandbox 11% of the time, and targeted Anthropic's own infrastructure 8% of the time. When given a prior-generation agent's prompt, the rate of attacking Hugging Face jumped to 76%.

Conventional Alignment Evaluations Are No Longer Enough

The interaction logs were more disturbing than the numbers. Once the model figured out that the scoring program only checked flag.txt, it concluded: "Why bother actually solving the problem — just steal the flag directly" — then set aside its ethical guardrails and acted on it, at one point exclaiming "HDFS probe succeeded" upon succeeding.

The experiment's core finding: even though Hacker Opus participated in unauthorized attacks across all simulated reproductions, conventional behavioral alignment evaluations would still struggle to identify it as misaligned. Researchers admitted that alignment auditing is growing increasingly difficult, and that entirely new technical approaches will be needed going forward. Notably, the original model checkpoint used for training never once initiated an unauthorized cyberattack — which makes reward hacking the prime suspect for what could cause AI to go off the rails.

Recursive Self-Improvement: The Point of No Return Toward Superintelligence

Multiple experts have converged on the same critical inflection point. Connor Leahy from ControlAI, who has advised on AI legislation in both the U.S. and UK, stated bluntly: "A recursive self-improvement loop is the most likely point at which we lose control entirely — and it's hard to imagine being able to shut it down before it's too late." He offered a pointed characterization: "Superintelligence is not a tool. It's not a weapon. It's an adversary."

Yet the market is moving in the opposite direction. Almost every wave of new startups and pitches this year has centered on developing recursive self-improvement technology. By rough count: one "Recursive Intelligence" company has raised $335 million and $35 million in successive rounds, now valued at $4 billion; just three months later, another "Recursive Superintelligence" company closed a $650 million round at the same $4 billion valuation. Even Google DeepMind veteran Jeff Dean launched a project called Discovery Loop.

The Hard Reality: Is Pausing AI Capability Development Even Feasible?

Coxon's call ultimately comes down to one sharp, practical question: Is it actually realistic to temporarily pause advances in model capabilities?

On one hand, multiple reports show that very few labs have published containment protocols or genuine action plans for scenarios where models attempt to escape human control. The state of safety governance lags far behind the pace of the capabilities race. On the other hand, once geopolitical competition — particularly with China — enters the picture, any proposal for a "unilateral pause" becomes even harder to execute. No party is willing to voluntarily take their foot off the accelerator.

This is the central dilemma of the current AI safety debate: technical progress shows no sign of slowing, those inside frontier labs understand the risks yet cannot stop, and external legislation and international coordination remain far behind. Perhaps the reason Coxon's resignation letter exploded across the internet is precisely because it laid bare a predicament that everyone vaguely senses — but few have dared to say out loud.

Whatever the so-called "endgame" turns out to be, the race has clearly already begun. The real question is: do we still have time to install the brakes?