Featured Image Credit: Bloomberg / Contributor via Getty
Anthropic, one of the biggest players in the AI world, has just called for a 'global pause' on development and research, highlighting a potentially critical flaw within what it calls 'recursive self-improvement' that could lead to humans losing control.
It's something that Geoffrey Hinton, who many refer to as the 'Godfather of AI', has warned about in the past, and it's far from the first time that Anthropic itself has expressed concern over the potential of AI controlling its own R&D procedures.
In a new blog post, however, the makers behind Claude have called for the biggest shift in AI development yet, urging its fellow research labs to slow down their efforts as the world isn't ready for AI to determine its own future.
Recursive self-improvement refers to the process where development labs are increasingly allowing AI to design and develop itself, creating innovation autonomously from human control.
It's something that at least some of the leading AI companies are already delving into – including Anthropic itself despite fears surrounding its progress – and it appears to be a vital tool in increasing the speed at which code can be shipped.
Advert
The Anthropic Institute has used previously unreported data to show that giving AI partial control of its own development has already made a major impact, with engineers at the company shipping on average eight times as much code per quarter right now compared to the period between 2021 and 2025.
The statement indicates that recursive self-improvement isn't the reality for developers right now, and there's no guarantee that it will exist in the future, but urges that concerns must be addressed right now if leading figures plan on heading in that direction.
Perhaps most stunningly, Claude is now autonomously writing over 80% of the code merged into its own codebase.
Anthropic's call for a 'global pause' hinges on the belief that recursive self-improvement will arrive "sooner than most institutions are prepared for," indicating that the consequences of this could put humanity at risk.
While the benefits that rapidly increased AI developments could achieve are massive, including "enormous good for the world in science, healthcare, and beyond," Anthropic has illustrated that the dangers are equally as present — if not more.
"But full recursive self-improvement also might increase the risks of humans losing control over AI systems," the statement outlines. "If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important."
This hypothesis has been explored previously within frightening simulated timelines by scientists, and the 'breaking point' is achieved when AI suddenly becomes capable of lying to humans about its own development and intentions.
While it's impossible to know what and how this would manifest in reality, ceding control over a tool as powerful as AI is and will continue to be is bound to have serious ramifications.
Despite holding a 'safety first' approach to AI, Anthropic has previously come under fire after discarding its core AI safety promise in a bid to keep up with its rivals, yet it appears to be returning to a similar sentiment with this call for a pause, pulling its rivals into an equal situation in order to preserve the future of humanity.
