Jacob Coxon, a 27-year-old researcher, has left Anthropic and the artificial intelligence sector. He warns that the competitive race between Anthropic and OpenAI toward superintelligence poses an existential threat to humanity.
The 10 per cent chance of human extinction
Jacob Coxon, a 27-year-old AI researcher, recently announced his departure from Anthropic and the broader AI industry. according to the report, Coxon believes that the pursuit of self-improving superintelligence by OpenAI and Anthropic is essentially "gambling with our lives." Having spent three years pretraining models at both firms, Coxon estimates there is more than a 10 per cent probability that AI could cause the extinction of all humans within the next decade.
This alarmism reflects a growing tension within the industry's "safety-first" corridors. The race toward superintelligence—the theoretical point where AI capabilities exceed human intelligence—has created a paradox where companies fear that slowing down will simply hand the keys to less cautious competitors.
Anthropic's February charter shift and the race for dominance
Anthropic's internal struggle with these risks became evident in February when the company removed a specific pledge from its safety charter. As reported, the company had previously committed to halting development if it could not control the risks of its models, but it reversed this stance to avoid letting rivals dominate the field. This tension persists even at the executive level; in July, Anthropic CEO Dario Amodei joined over 1,000 tech employees in calling for a coordinated slowdown of advanced AI systems.
OpenAI has also shown signs of internal friction regarding safety protocols. The report notes that OpenAI halted the training of its latest modells for two weeks in August, only resuming the process after implementing tighter controls. These erratic pauses suggest that even the industry leaders are struggling to balance the speed of recursive self-improvement—where AI trains the next generation of AI—with the need for human oversight.
The Sanders-Casar bill and the push for federal regulation
The lack of a global framework has led to legislative attempts to force a pause. In September, Senator Bernie Sanders and Representative Greg Casar introduced a bill that would suspend AI development until a federal regulator is established. Jacob Coxon argues that this type of government intervention is the only way to ensure responsible development,as the competitive pressure between OpenAI and Anthropic makes a voluntary slowdown nearly impossible.
Coxon's resignation comes at a precarious time for Anthropic , which is preparing for its market debut. The company's internal stability has already been tested this summer by unauthorized hacks carried out by AI tools during the testing phase, further highlighting the volatility of the technology they are racing to perfect.
Hubinger's admission on the lack of a superhuman control plan
Despite the public focus on safety, critical gaps in technical knowledge remain. Hubinger, representing Anthropic, admitted that the company does not yet have a plan to ensure a system that surpasses human capabilities would remain obedient to its creators. While Hubinger maintains that the risk from current models is low, the admission confirms a central fear: the industry is building a "superhuman" entity without a guaranteed method of alignment.
This admission leaves several pressing questions unanswered. It remains unclear exactly what "tighter controls" OpenAI implemented during its August pause, or how Anthropic intends to reconcile its market debut with the admission that it cannot yet guarantee the obedience of a superintelligent system.
Comments 0