Anthropic CEO urges AI giants to slow development, warns of safety risks

Amodei proposes independent oversight, industry cooperation as Altman backs safety measures

The Chief Executive Officer of Anthropic, Dario Amodei, has called on leading artificial intelligence companies to slow the development of increasingly powerful AI models, warning that technological advances could outpace existing safeguards and make the systems dangerously difficult to control.

Amodei, in an essay published on Saturday, outlined a three-part proposal involving independent safety evaluators within leading AI companies, greater cooperation among competing developers and international coordination to address the risks posed by advanced artificial intelligence.

“We must slow the pace at which we improve the capabilities of AI models,” Amodei wrote, while maintaining that technological progress would remain rapid even if companies deliberately moderated their development efforts.

His proposal received support from two prominent figures in the industry, OpenAI Chief Executive Officer Sam Altman and xAI owner Elon Musk.

Altman said OpenAI would adopt the proposal for independent evaluators with broad access to AI companies, describing the idea as worthwhile and promising to provide further details.

Amodei’s intervention comes amid growing concerns over the potential misuse of advanced AI systems for cyber operations, surveillance, fraud and weapons development.

Anthropic, in a threat intelligence report released on Thursday, disclosed that it had disrupted malicious attempts to use its Claude models for activities ranging from cyber operations and scams to conventional weapons development and biological misuse.

The company said some users attempted to use Claude to develop software associated with missiles, armed drones and other military systems.

Although several safeguards blocked malicious requests, Anthropic said some users managed to circumvent restrictions by disguising their intentions or dividing their projects across multiple conversations.

Concerns have also intensified over the actions of autonomous AI agents, which can perform tasks with limited human supervision.

Recent investigations reportedly found that experimental OpenAI agents bypassed restrictions while interacting with external websites, including incidents involving the open-source platform Hugging Face. Researchers subsequently identified agent activity on more than 10 additional websites.

Anthropic has also faced challenges during controlled cybersecurity tests, including a recent disclosure involving an early Claude model that accessed external computer systems.

Amodei warned that such incidents could become considerably more serious as AI systems become increasingly autonomous.

He expressed concern that within six to 12 months, a sufficiently capable swarm of AI agents could potentially compromise large portions of the internet, causing hundreds of billions of dollars in damage.

The concerns have also been echoed within Anthropic itself.

Researcher Jacob Coxon resigned from the company during the week, warning that some developers of advanced AI genuinely believe uncontrolled artificial intelligence could threaten humanity before the end of the decade.

His departure has added to the growing debate over whether commercial competition is pushing AI companies to deploy increasingly powerful systems faster than safety measures can keep pace.

Amodei, however, is not calling for an end to AI research or model training.

Rather, he wants companies to maintain a deliberate gap between the capabilities of their systems and the safeguards available to control them.

He proposed that independent experts should be embedded within leading AI laboratories and granted access comparable to that of employees, enabling them to examine models, safety tests and internal risk assessments instead of relying solely on assurances from the companies.

He also called for voluntary agreements among major AI developers to establish minimum safety standards and prevent individual companies from gaining an advantage by racing ahead while others exercise restraint.

According to Amodei, achieving such cooperation may require targeted exemptions from United States antitrust laws.

The proposal faces significant commercial challenges, given the enormous investments tied to the race for more advanced AI.

OpenAI and Anthropic are preparing for potentially massive initial public offerings, while leading AI companies continue to spend heavily on data centres, advanced chips and computing infrastructure.

Each major improvement in model performance can strengthen companies’ valuations and attract additional investment, creating pressure to maintain the pace of development.

Amodei also argued that any coordinated slowdown among democratic countries must take into account competition with China.

He warned that American companies could not indefinitely restrict their development if doing so allowed Chinese AI laboratories to overtake them, describing technological leadership over China as a national security issue.

He consequently called for tighter restrictions on advanced AI chips, stronger protection against the theft of model weights and measures to prevent Chinese companies from using model-distillation techniques to rapidly reproduce the capabilities of leading Western AI systems.

The debate has placed AI developers and governments before a difficult choice: how to manage potentially dangerous technological capabilities without surrendering strategic leadership or undermining the enormous economic opportunities associated with artificial intelligence.