Anthropic says AI could build its own successor, calls for slowdown in development

Anthropic, the artificial intelligence (AI) company behind Claude, has called on leading AI firms to consider slowing the development of advanced AI systems. The company said AI is progressing so rapidly that these systems may soon be able to improve themselves without human intervention.

Anthropic warned that this could eventually lead to a situation in which AI designs, trains and improves its own successors with little to no human involvement.

On Thursday, 4 June, Anthropic shared a post on X (formerly Twitter), writing, “Our internal data shows Claude is accelerating AI development—a possible path to recursive self-improvement, or AI autonomously building a more capable successor. It’s happening faster than we thought, and the implications deserve greater attention.”

In a blog post, the company added, “Taken far enough, and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor. This is called recursive self-improvement. We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for.”

According to Anthropic, its AI assistant Claude is already helping engineers write most of the code used to develop the company’s AI systems.

The company said this has significantly increased productivity and accelerated research. Its internal data shows that AI systems are becoming increasingly capable of handling longer and more complex tasks.

As a result, Anthropic believes these improvements could eventually lead to AI playing a major role in creating future models.

“This could revolutionise society—medicine, technology and the economy—for the better. But it may also compound alignment issues and ultimately lead to loss of control,” Anthropic added.

The company said the world should have the option to temporarily pause or slow frontier AI development if the technology begins advancing faster than safety measures and regulations can keep pace.

“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” the post read.

Anthropic also proposed an international agreement that would allow governments and AI companies to co-ordinate any future slowdown. It suggested establishing a verification system to ensure all participants comply with the rules.

The company added that, in the coming months, it will organise discussions to answer questions, particularly regarding the prospect of fully recursive self-improvement in AI systems.

More details here...