Anthropic, the world's largest artificial intelligence startup, has called for a global freeze on AI development and warned that humanity risks losing control of the technology.
The company that developed the chatbot Claude has offered to suspend work on more powerful systems if it receives assurances that others will do the same.
«"If there were a way to effectively slow down the development of this technology to give ourselves more time to address its enormous challenges, we believe that would probably be a good thing," the company's executives wrote.
They compared the emergence of powerful artificial intelligence systems to an "arms control problem" and warned that time was running out to rein in the technology.
Anthropic recently overtook ChatGPT developer OpenAI to become the world's most valuable AI company, with a market capitalization of $965 billion (£719 billion). It has delayed the public release of its most powerful system, Mythos, due to concerns that it could be used to conduct destructive cyberattacks.
In an essay published Thursday evening, Marina Favaro and Jack Clark, Anthropic's head of research and president, said artificial intelligence technology is approaching the point where systems will begin to evolve on their own.
They said this moment, known as "recursive self-improvement," could lead to an explosion of capabilities, but could also mean that humans would be unable to control AI systems.
Anthropic positions itself as the most committed to ensuring the safety of this AI technology, while simultaneously making scientific breakthroughs that it says could lead to the creation of dangerous systems.
The company's CEO, Dario Amodei, said there was a 25 percent chance that "things will go very, very badly.".
On Thursday, the company said: "If slowing growth allows the less cautious market participants to simply catch up technologically, it could make everyone less secure.".
«"We believe that it would be beneficial for the world to be able to slow down or temporarily pause the development of advanced AI technologies to allow social structures and process coordination research to keep pace.".
However, the report warned that coordinating a global freeze would be extremely challenging, pointing to fierce competition between the US and China in artificial intelligence.
«"A significant slowdown or suspension would require the agreement of several well-resourced laboratories at or near the cutting edge of research, in different countries, to cease operations under the same conditions," the document says.
«"It would also require that everyone could confirm that others had actually stopped... Practice runs are much easier to hide than missile silos.".
Ms. Favaro and Mr. Clark said Anthropic employees are now producing eight times more code than they did between 2021 and 2025, and that the AI is improving its skills at proposing new ideas and planning research, which they say is a step toward creating self-organizing AI.
Other experts argue that the company is exaggerating AI capabilities or creating AI warnings to introduce rules that could harm competitors.
This comes after a massive sell-off in shares of companies developing microchips for artificial intelligence after semiconductor maker Broadcom disappointed investors with its sales forecast.
The company's shares fell 13%, wiping about $300 billion off its market capitalization, with shares of other chipmakers such as Micron, SK Hynix and Arm also falling sharply.
The sell-off has sparked concerns about a bubble in the artificial intelligence market due to questions about demand for Broadcom chips.
Try The Telegraph today for full free access. Discover their award-winning website and essential news app, plus helpful tools and expert guides on finance, health, and leisure.