He pointed out in his essay that AI had advanced “drastically faster” including its “ability to build the next generation of AI” – and mentioned an incident involving rival OpenAI which has revealed that agents had hacked, external targets they were not asked to attack in July.
Amodei called for “building AI at a balanced rate that aims to ensure its safety while still achieving its benefits”.
This would not mean “halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this”.
He was committing Anthropic to this “unilaterally” – as well as calling on governments “to require other frontier companies to match”.
Amodei said he recognised that regulation might not be able to keep up with the pace of AI, and therefore called on AI companies to “voluntarily work together to set standard” in parallel with those regulation.
The Anthropic CEO went on to address the impact that a slowdown would have on the industry and competition with leading developers worldwide, particularly China.
“I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong,” Amodei said.
This would have to be done in a co-ordinated manner “without sacrificing commercial advantage or the United States’ lead in AI”. Any slowdown would have to be limited, he said, to avoid allowing China to pull ahead.
He urged the US government to take measures so that US companies’ AI chips could not be sold to China – or the technology shared with authoritarian countries.
His former employee, Jacob Coxon, told the BBC, however, that the initiative to slow down must go beyond the US – “there’ll need to be some sort of co-ordinated slowdown with China if we’re going to avoid a race at an international scale”.
Amodei’s post has prompted a wide range of responses.
Clement Delangue, the CEO of the AI platform Hugging Face, said he was launching a new project called the Open Alignment Initiative, adding that he wanted to be among “embedded evaluators” that Amodei proposed could be part of a solution.
Hugging Face was hacked by OpenAI agents earlier this year, prompting outcry over AI safety.
“Let’s make AI safer by making it more transparent,” Delangue wrote on X.
Musk, who also voices support for Amodei, once called Anthropic “evil” but has changed his tone since signing a $15bn (£11bn) deal to sell computing capacity to Anthropic in May.
However, some observers suggested that Amodei’s post was less about safety than about consolidating control over AI technology.
“Dario makes the case to stop open source and concentrate enormous technological and economic power with Anthropic,” wrote Chamath Palihapitiya, an investor and co-host of the tech podcast “All-In”.
Notions of slowing down or even pausing AI development have long been met with cynicism in certain corners of Silicon Valley, with critics accusing leading AI developers of hyping their technology as a marketing ploy.
Anthropic and OpenAI are both reportedly preparing for potentially record-setting initial public offerings.
