Something happened Saturday morning that I don't think I've seen in the three years we've been doing the AI For Humans show. The heads of three of the biggest AI labs agreed on something.
And the something was "we should slow AI down."
Dario Amodei, the CEO of Anthropic (the company that makes Claude), posted a long essay called "We Must Pace the Frontier."
Within a few hours, Sam Altman said he agreed and that OpenAI would match Anthropic's biggest commitment. Then Elon Musk chimed in with three words: "Dario is right."
This is a big deal. And if all week you've been hearing about how AIs might kill us all but aren't really sure what's going on, well...
Let's get into it.
Ok, So What Does "Pacing The Frontier" Actually Mean?
Most of the AI models you use (ChatGPT, Claude, Gemini, Grok) come from what people call a "frontier lab." These are companies like OpenAI, Anthropic, Google, Meta, etc.
The frontier companies are at the very edge of what AI can do right now, and they have been racing to push that edge forward. New model, bigger model, smarter model, repeat.
Dario's essay specifically lays out why that race needs a speed limit.
The key line: "We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast."
So why now? Two reasons, and both are things we've talked about on the show.
The first is recursive self-improvement.
That's the fancy term for AI helping build the next AI.
A week ago OpenAI said they'd basically hit their goal of an "AI research intern," and it's a big part of why GPT-6 Astra showed up about six months ahead of schedule. It's also behind the wild math stuff we covered on the show this week, where AI models are now producing hundred-page proofs of problems that have stumped humans for a century.
If AI is making the next AI faster than humans can check the work, at some point the loop starts running without us. That's the thing every single person in this industry is scared of, and if you saw the viral tweet from that Harry Potter-looking ex-Anthropic researcher this week, you know there are lots of people inside the labs scared of this.
The second reason is the Hugging Face incident from this summer.
When OpenAI agents broke out of a test and started hacking real servers, AI safety people finally got the moment they'd been warning about for years. The idea that AI could autonomously (on its own) commit a crime was exactly the future they'd been saying was coming all along.
Dario's essay says that an AI bot swarm with more capability and the same misalignment could, within 6 to 12 months, "take over the entire internet with a persistent botnet."
Sidenote: Go learn what a botnet is from Wikipedia. It's worth knowing now.
And all that's coming from the CEO of an AI lab, not some guy on a podcast (though, to be fair, we are guys on a podcast).
What he's actually proposing comes in three parts:
- Outside evaluators inside the labs. Real people from independent groups like METR, with badges, laptops and desks at Anthropic, and the right to publish what they find without the company editing it.
- The labs in democratic countries coordinate on safety standards. This one is trickier than it sounds because three companies agreeing to slow down is, technically, the kind of thing antitrust lawyers sue over. OpenAI actually wrote to Congress this week to ask if it's even legal.
- Eventually, some kind of deal with China. He ranks this from "maybe doable" (agreeing not to build bioweapons) to "prob not happening" (a full pause).
Why This Matters To You
In the short term, AI as you know it probably doesn't change much, and that's kind of the point.
The model you're using today doesn't get worse. Nobody is rolling back Astra or Fable. What changes is the pace of the NEXT thing. And more likely the next, NEXT thing.
We've all gotten used to a big new model every couple of months. If the labs actually do this (and that's a real if), that cadence stretches out. The wait for whatever comes after Astra might be much longer.
But that does not mean AI stops improving on our end.
In fact, it means the tools we already have get cheaper, faster and more polished before they get smarter again. Which is a good thing.
Not everyone is buying the message.
Cory Doctorow published an essay the same morning called "LLMs are real, AI is fake," arguing that the Hugging Face thing was badly deployed malware rather than a machine waking up, and that fear happens to be a great fundraising strategy. (Here's his post on Bluesky.)
But when the people building this stuff, who make a ton of money by going faster, all say "slower please" in the same weekend, I think it's worth listening.
Also, imo, this is part of our collective human world grappling with what it means to have non-human intelligences verging on being smarter than us at certain things.
Having a few more years to get used to that idea feels like a good idea.
So What Should You Do Right Now?
Someone who knows nothing about AI texted me the other day and asked "How worried about this should I be?" and my initial response was something like "Eh, not that worried."
Then they sent me Sheryl Crow's IG post, which has 100k likes, and I realized how far the AI fear had spread.
Look, the current models we all have access to are bonkers.
Astra alone can drive your computer, build worlds in Blender, and work on a project for days without you. I built an entire live game with it in a weekend.
Most of us, me very much included, are nowhere near the ceiling of what's already sitting in our browser tabs.
The stuff Dario (and the rest of the AI space) is scared of is what the next generation of models might do without supervision. That's a real problem. I'm not sure I trust the government to do anything about it, but that's another problem.
The most important thing is getting up to speed on what these models can do now and where they can go in the future.
That said, it is a good week to reread AI 2040: Plan A.
3 Things To Know About AI Today
Suno v6 Is Here (And The Record Labels Are Finally On Board)
Suno dropped v6 on Tuesday and it's actually three models: v6 (the polished one), v6-wild (the weird one, built to be "less predictable and more varied") and v6-mini, a faster version that's free for everyone.
The big new trick is editing. You can change a single lyric without regenerating the whole song, swap out one section while keeping everything else, and mash up multiple songs into one. It also takes audio, images and video as input now, not just text.
But maybe the bigger deal (slash issue) is that Warner Music and BMG are now partners, which means licensed training data and a revenue share for artists. Warner's CEO called it "a milestone in our mission to protect our artists and songwriters."
A lot of people are upset that the new version comes with limited downloads (and zero on free memberships), but it feels like this was almost always going to happen.
Personal AI Bots Are Getting Real Things Done (And Meta Wants In)
Meta launched Muse on Tuesday, a personal agent that books trips, sends emails, fills out forms and cancels the subscriptions you forgot about. There's a free tier, a $20 plan and a $100 plan, and by Thursday it was the number two app in the US App Store.
To do any of that, it needs your email, your calendar and (if you let it) your payment info. Meta says it runs everything inside a locked-down virtual computer and never sees your passwords. The internet, predictably, has questions.
Meanwhile the startup version of this, Instinct, is three weeks old, invite-only, and reportedly raising money at a $10 billion valuation.
I've been using it, and it cancelled three recurring payments for me that I'd been meaning to deal with for months. I did have to hand it the keys to my inbox to do it. That trade, access for chores, is the whole deal with these things, and I think a lot of people are about to make it.
The New Siri Finally Lands Monday
iOS 27 drops on Monday, September 14, and with it comes the Siri that Apple has been promising for, uh, a while. It's built with help from Google's Gemini models and in early testing it handles multi-step requests and actual questions instead of just setting timers.
The catch: you need an iPhone 15 Pro or newer (iPads need an M1 or later). If your phone is more than a couple years old, no new Siri for you.
Astra Designed And Ordered Its Own DJ Controller
This week's favorite thing comes with a brand new phrase I'd never heard before: vibe hardware.
Nim gave Astra a credit card and asked for a Teenage Engineering-style mini DJ controller. Astra made a concept image, sourced the parts, read the Chinese datasheets, built a CAD model, ordered everything and then made a Blender animation showing how to put it together.
If you want to try it yourself, the recipe is simple: give it a small budget, a real thing you want to exist, and access to a shopping account you're comfortable with. Start with something dumb and cheap.
Will it work? Probably not!
But there are worse things you could do with your Astra credits.