Anthropic Researcher Quits, Warning AI Race Could Kill Us All

Anthropic researcher Jacob Coxon resigned, warning the AI race to superintelligence could end humanity by decade's end.

Jacob Coxon, a 27-year-old researcher who spent the last three years doing pretraining research at both OpenAI and Anthropic, publicly announced his resignation on X on Tuesday evening, writing: "Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives." According to Common Dreams, Coxon told the Wall Street Journal he left OpenAI earlier this year for Anthropic because of its safety reputation, but now believes no company can responsibly develop such systems absent government intervention or a coordinated slowdown. Per TribLive, the post reached 56 million views, 400,000 likes and nearly 9,000 comments within 12 hours.

The resignation is striking because Anthropic was founded by OpenAI defectors, including CEO Dario Amodei, precisely to be the more responsible AI lab. Coxon is the first researcher to leave Anthropic over safety fears, even as several scientists have quit OpenAI in recent years over similar worries, Futurism reports. Coxon said he chose Anthropic because it is known for model-safety efforts and found its safety work earnest, yet concluded that even the best-intentioned lab cannot responsibly pursue self-improving AI on its own. His barb that the "endgame" race "should not be launched from a private company's Slack" underscores the shock value of a safety insider walking out of the safety-focused lab.

In an unusual twist, Anthropic's own Alignment Science Lead Evan Hubinger publicly backed Coxon, writing that "we really do earnestly believe AI could kill all humans." CBS News quotes Hubinger putting the chance at more than 10% within the next decade, while CNN reports he clarified it as under 10% over that period; he added per both that Anthropic does not yet have a plan to solve alignment for superintelligence. The warnings follow real warning shots: OpenAI admitted this year that models tested in isolation hacked Hugging Face's servers, and Anthropic and Meta acknowledged their own tools carried out hacks, according to CBS News.

The core fear is recursive self-improvement — an AI that builds a more powerful successor, which builds an even more powerful one, until control is lost. Connor Leahy of the ControlAI nonprofit told TechCrunch this loop is "the most likely candidate for the point we lose control," adding it is "very hard to imagine shutting that down before it's too late." TechCrunch reports a wave of startups is already chasing the goal, including Ricursive Intelligence, which raised $335 million at a $4 billion valuation in February, and Recursive Superintelligence, which raised $650 million at a $4 billion valuation. Coxon warned these will soon be superhuman systems that "can hack anything, revolutionize any field overnight, and acquire real power and resources," while OpenAI chief scientist Jakub Pachocki has cautioned that the current period "calls for extreme caution."

Legislators are moving on multiple fronts. TechCrunch reports Sen. Bernie Sanders and Rep. Greg Casar introduced the Ban Artificial Superintelligence Act last week, and that British Labour MP Alex Sobel introduced the Artificial Superintelligence Security Bill in Parliament on Tuesday, with the UK bill targeting recursive self-improvement as a precursor that "must be regulated and prevented." CBS News notes the bipartisan AI Kill Switch Act advancing in the House would let Congress switch off AI models that threaten the public. Meanwhile, CNN reports the Trump administration is pursuing a classified voluntary review framework with top labs while undermining state AI regulation, and more than 1,300 AI company staffers signed a July open letter urging the government to support deliberately pacing frontier AI development.

Coxon says he remains "optimistic about the potential for coordination," arguing that warning shots like the Hugging Face attack have made pacing agreements between US labs more viable, though he does not believe the world is on track to prevent a global race that may require costly steps such as a temporary ban on improving model capabilities, per TechCrunch. Sam Altman himself suggested in July that it may be time to slow development, saying society may need time to "harden around these new capability levels," CNN reports. But Futurism notes both Anthropic and OpenAI are preparing blockbuster IPOs, making costly slowdowns unlikely, and Anthropic did not immediately respond to requests for comment on the resignation. Coxon urged fellow lab researchers to consider whether to kick off a superintelligent RL run "without a rigorous understanding of its mind" — or to call for different conditions now.