I appreciate an AI executive saying the industry needs to slow down. These companies are selling us increasingly capable systems while asking us to trust that they can keep them under control. Hearing someone question the pace is welcome, especially when his own company has money to make from pushing ahead.
But Anthropic CEO Dario Amodei helped ring this bell. Now that he is worried about where the noise is taking us, he is asking others to help contain it. His safety concerns deserve attention, but I think his proposed solution puts too much faith in managing a competition that no single company or government controls.
In his We Must Pace the Frontier essay, Amodei calls for slower advances in AI capabilities so safety work can catch up. He cites AI helping develop subsequent systems and incidents of agent misbehavior, including at Anthropic.
His warnings are serious, and I would rather hear them than another sales pitch about how AI will fix everything. Acknowledging that your own industry could create dangers it is not prepared to handle is worthwhile, even when that admission invites uncomfortable questions about the products you are building.
The problem is that you cannot unring a bell. You can change what you do next, but you cannot make competitors forget what they have learned or abandon what they hope to gain. Once the race is underway, asking everyone to proceed carefully becomes a much harder sell.
China is the obvious test. If American companies slow down and Chinese competitors keep advancing, how much ground would the US be willing to surrender in the name of safety?
Amodei does address this. He says pacing within democracies is limited by their lead over China, and advocates chip restrictions, measures against smuggling, and protection against model theft to preserve that advantage.
That is a strategy for buying time, but it is not a guarantee that the time will be available. It depends on restrictions working well enough and policymakers correctly judging how far ahead they are. What happens if they discover that the gap is smaller than they thought?
There is an uncomfortable tension here. If AI could become uncontrollable, keeping the most powerful systems in American hands does not automatically make them safe. An American company could win the race and still produce something it cannot reliably control.
Amodei proposes embedded independent evaluators, coordination among democracies, and eventual global agreements. Anthropic commits to the evaluators, but the essay sets no specific unilateral speed limit.
The outside scrutiny deserves credit. Giving independent reviewers meaningful access could make safety promises more accountable, particularly if they can expose problems that a company would prefer to keep out of public view. That is a concrete idea worth pursuing.
But identifying a danger and accepting the competitive cost of addressing it are different things. A reviewer could recommend caution just as a rival announces a more capable system. That is when we would learn how much influence the reviewer actually has.
I do not think Amodei is naive because he assumes China will happily cooperate. He recognizes the difficulty. The naivety is in how much his proposed path depends on governments and companies managing that competition without letting it overwhelm their restraint.
None of this means we should give up on safety. You cannot unring the bell, but you can still establish rules for what happens next. Those rules need to work when cooperation breaks down, because that is when they will matter most.
What would Anthropic refuse to build even if a competitor went ahead? Which safety finding would justify accepting a disadvantage, and who would have the authority to enforce that decision?
I appreciate Amodei making the case for caution. But the test is whether that caution survives the possibility of losing, not whether everyone agrees that slowing down sounds sensible.
Update: Sam Altman has now backed Amodei’s call to pace AI development, saying the subject has been a primary focus of recent discussions at OpenAI. He also says OpenAI will commit to independent evaluators with employee-like access, with more details coming soon.
That gives Amodei’s proposal support from a direct competitor, and both companies deserve credit for committing to outside scrutiny. But Altman’s post does not announce a specific slowdown or explain what OpenAI would do if safety required accepting a competitive disadvantage. Two American AI companies agreeing on oversight is encouraging, but it still does not answer the question about China.
Support independent tech journalism
NERDS.xyz is independently owned and operated. If you enjoy my coverage of Linux, AI, hardware, cybersecurity, and tech culture, consider supporting the site on Ko-fi.
Support NERDS.xyz