Anthropic researcher announces resignation over fears company is 'gambling with our lives' and 'acting iresponsibly'

Jacob Coxon, a pretraining AI researcher who has worked at both Anthropic and OpenAI in the past few years, has announced his resignation from the former after claiming the leading development lab isn't 'acting responsibly' by 'gambling with our lives'.

He's far from the only individual who has departed the AI industry over ethical and moral concerns, but his admissions serve as a frightening warning for our future as development continues to speed towards potentially dangerous 'superhuman' models.

Not only are the dangers of these advantage AI developments unveiled by Coxon – shocking many in the process – but he also indicates that the companies behind these models are well aware of the risks yet continue their efforts, raising the risk of civilization-ending dangers.

These predictions mirror the frightening timeline outlined in a recent scientific study, and echo the worries of some of the AI world's most prominent figures — and seemingly nothing is done to stop the tech from progressing towards a dangerous future.

Why did Jacob Coxon resign from Anthropic?

While Coxon doesn't explicitly give a reason for his resignation, the information he outlines is presume to be a key motivating factor for why he departed from the company currently at the forefront of the AI revolution.

He suggests that "many executives and senior researchers will couch their phrasing in the process to sound sensible – but I hear the same people express fear privately," adding that "no other human activity poses this level of danger," suggesting that the disparity between fear and action drove him to depart.

Anthropic specifically is aware of these risks according to Coxon, "but they are locked in a race to get there first – they believe no one else will act responsibly, so they must do it themselves, despite the risk."

This matches what Anthropic itself has outlined, as it previously abandoned its core AI safety promises in order to not be left behind in the industry, using the so-called necessity for responsible action as justification for involving itself in the risk.

Coxon specifically calls out the leading development labs, proclaiming that "accepting this race and entering the 'endgame' is a hubristic gamble that should not be launched from a private company's Slack.

"Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available," he continued.

How could AI 'kill us all' in the future?

One of the biggest questions that might emerge from Coxon's message is how AI could lead to the destruction of humanity, as for some that might seem impossible for what's 'only' software at this point in time.

As Coxon warns, we should "not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources."

From a purely human perspective, AI will imminently give criminals an incredible amount of power with real world consequences, such as hacking the world's biggest companies and government institutions, or launching fatal bioweapons across the globe.

Giving criminals or foreign adversaries access to the military, for example, is likely to have devastating consequences — and AI already has the potential for this power to be realized.

Beyond this there are also the risks that arrive once is able to fully automate its actions, as we suddenly run the risk of losing 'control' and become vulnerable to deception if it wishes to gain power over us, and that's a door that many – including Coxon – believe simply cannot be opened without the proper preparations.