Jacob Coxon, a 27-year-old researcher, has resigned from his position in pre-training AI models at Anthropic. Coxon claims that the current corporate competition to develop superintelligence poses an existential risk to human civilization.
Why Jacob Coxon Views Superintelligence as a Greater Threat Than Nuclear War
Jacob Coxon spent three years conducting pre-training research at both OpenAI and Anthropic, and he now asserts that neither organization is acting responsibly. According to the report, Coxon believes the relentless corporate sprint toward "self-improving superintelligence" is a gamble with human lives that could outpace the dangers posed by global climate change or nuclear conflict.
Coxon argues that while executives and senior researchers may use sensible language in public press releases, they express deep fear privately. He suggests that the internal belief that AI could cause a total collapse by the end of the decade is not a marketing tactic, but a genuine concern shared by those closest to the technology.
Evan Hubinger’s 10% Probability of Human Extinction
The warnings from Coxon are echoed by other high-level staff within the company. Evan Hubinger, the Alignment Science Lead at Anthropic,admitted on X that he and his colleagues believe AI could potentially "kill all humans." Hubinger specifically estimates that there is a greater than 10 percent chance of such a catastrophe occurring within the next decade.
As the report notes, Hubinger acknowledged that Anthropic currently lacks a viable solution for aligning superintelligence. while he believes the company is trying its best, he stated plainly that Anthropic is not clearly on track to solve the alignment problem, leaving a dangerous gap between the capability of the models and the ability to control them.
The Paradox of Dario Amodei’s "Doomer" Label
This interal crisis highlights a striking contradiction at the top of the organization. Anthropic CEO Dario Amodei has been labeled an industry "doomer" due to lengthy blog posts expressing deep existential anxiety about the technology he is leading his company to create. This suggests a corporate culture where the leadership harbors dread about the very product they are racing to finish.
This tension is part of a broader industry pattern. Leaders from Meta, OpenAI, and Anthropic have recently acknowledged that the intense competitive pressure makes it impossible for any single company to pause development unilaterally. Instead, these firms are calling for international governance and support from the US government to create a framework that buys time for safety research.
The Feasibility of a Temporary Ban on AI Capability Improvements
To halt the current momentum, Jacob Coxon has urged his peers to speak out and called for a temporary ban on improving model capabilities. He argues that such a drastic measure is necessary to force a shift toward international coordination and safety controls, as the current trajectory is not on track to prevent a global race.
However, several critical details remain unverified. The report does not specify what exact technical "guardrails" Coxon believes are misisng, nor does it explain how a temporary ban could be enforced across different jurisdictions to prevent a "rogue" state or company from continuing development. Furthermore, it remains unclear if the US government is actively considering the international governance structures that Meta and OpenAI are requesting.
Comments 0