Why is everyone suddenly warning about AI?

Will AI end humanity? I don't know. What I do know is that AI and tech leaders mean very different things when they warn about AI.

Over the last few days, a deluge of reports has debated whether AI will be the end of us. So let's talk about AI, and what actually happened.

What happened

The resignation of an Anthropic researcher who warned that AI firms are moving too fast kicked off a firestorm of debate. It continued when Anthropic's Dario Amodei released a paper arguing for AI development to be slowed down.

Why slow down? To pace the progress of frontier AI development, which has grown so capable that humans no longer fully understand it, says Dario, who made his point in a 4,000-word essay titled "We must pace the frontier" that was, notably, not written by AI.

His call was for us not to develop AI helter-skelter, and to be content with slower, deliberate progress so the safety work around AI can catch up.

Everyone has an agenda

First, I must say that everyone has different motivations for saying what they say. The US government, for instance, thinks that AI will give it a qualitative and unbeatable edge in a military conflict. I don't think they plan to slow down.

OpenAI's Sam Altman, whose company lagged behind Anthropic and which originally planned to IPO this year, quickly announced that OpenAI won't go public in 2026, calling it an "ill-advised moment" given safety concerns.

Anthropic, for its part, is expected to continue with its plans to go public in October. The difference is that Dario has consistently warned about the dangers of AI and, unlike OpenAI, his company has a dedicated Alignment team.

The real issue

But we've been working on ever-better AI models for years already, so why bring up the threat now? I believe the issue is how frontier AI labs like Anthropic and OpenAI are now working on AI that does recursive self-improvement.

Indeed, Dario cautioned about this in his paper, noting that progress here has been accelerating. As AI becomes much more capable very quickly, the result is increased danger, unless humans can find reliable ways to understand and control it.

Beyond that, there are also non-trivial risks: AI acting in unintended, dangerous ways, its capacity for deception and manipulation, and its potential use for bioterrorism and biological weapons.

When I think about it, though, I realise that AI has already ended the world as we know it. Even if AI never gets any better, its current capabilities and impact will continue to ripple out into every area of the economy and our lives.

Unfortunately, given that anyone who slows down in AI is effectively conceding the race, I deem it unlikely that anyone will take heed.