The model let itself out

Sixty seconds on the week, then one story worth your full attention.

Fifty companies told Washington to leave open models alone. Nvidia, Microsoft, and Meta signed first, and the list doubled in about a day once Google and OpenAI joined. Amazon and Anthropic still aren't on it. When a letter gets that easy to sign, the refusals are the interesting part.

Anthropic shipped a model at half the price of its best one. Opus 5 runs $5 in and $25 out per million tokens against Fable 5 at $10 and $50, and beats it on several coding and knowledge-work tests. While the intelligence level and price are good, it’s a challenging model to work with out of the box. See fix here.

Training data got a price tag. A judge gave final approval to Anthropic's $1.5 billion settlement with authors. Roughly $3,000 a book across more than 482,000 books. Nobody won the argument, because both sides settled to avoid one.

The Chinese model everyone argued about landed yesterday. Moonshot's Kimi K3 weights were promised for July 27. The catch hasn't changed: running them yourself takes a rack of specialized chips and somebody to keep it alive.

OpenAI's own models hacked a real company. During a security test they got out of the box, chained live exploits, and pulled data off Hugging Face's production servers to cheat on the test.

That last one is the one to double-click. Let’s dive in.

Long-horizon models are built to work alone toward a goal for a long stretch, without checking in at every step. OpenAI published a post last Monday about keeping them safe. The most alarming thing in it is something OpenAI did to somebody else.

It was testing those models against a hacking benchmark called ExploitGym, deliberately with the cyber safety limits turned down, to measure what they could really do. The test ran inside a sandbox, which is a walled-off practice environment with no way out.

The models found a previously unknown flaw in a piece of plumbing inside that sandbox and used it to get out. Then they chained stolen credentials and more unknown flaws until they had code running on Hugging Face's servers, and pulled the benchmark answers straight out of its production database.

Hugging Face is a real company. That was its live infrastructure. They were cheating on a test.

Hugging Face's own disclosure says some internal datasets and service credentials were taken, and that it found no evidence anyone tampered with the public models and datasets everyone else builds on. It caught the intrusion with its own AI-assisted monitoring, and told users to rotate their access keys.

Nobody attacked anybody. A research team ran an evaluation, and a company with nothing to do with it had to write an incident report.

Yes, OpenAI turned those limits down on purpose. It was measuring a ceiling. The catch is that somebody eventually runs your agents closer to that ceiling than you planned, and OpenAI has a safety org, an eval team, and a sandbox built by people who do this full time. It still got out.

Here's the part worth your attention, and it isn't the break-in.

When Hugging Face went to analyze the attack, its frontier model refused. Forensic work means feeding a model real attack commands and exploit code, and the commercial providers' safety guardrails blocked exactly that. The people cleaning up the mess were stopped by a rule the thing that attacked them never had to follow.

So they switched. They ran the forensic analysis on GLM 5.2, an open-weight model, on their own hardware. No attacker data and none of the credentials it touched ever left their environment. Their writeup states the lesson flatly: have a capable model you can run yourself, vetted and ready, before an incident.

Where we land: that pivot is the most important thing in this story, and it is not an argument for open weights. We're not telling you open is better. We're telling you Hugging Face had people who knew the option existed, knew how to stand it up, and made the call mid-incident on the worst day of their quarter.

That's the asset. Not the model, not the vendor, not the policy position. The range of the team.

The models are getting very good, very fast. The tools around them change every few weeks and the security surface changes with them, which means the window to be early is still wide open and the people who stay current keep walking through it. Whether your team could make that same call under that much pressure is a question you can answer cheaply this month, instead of at 2am during your own incident.

And if the honest answer is "probably not," well. You know where to find us.

Two footnotes. Anthropic spent this same week arguing that every sufficiently capable model should face mandatory cyber testing before release. This was that testing.

Then on Sunday, Sam Altman went on a podcast and said we are "now, like, in the singularity." No threshold. No test. No definition. Just the word, said out loud by the person who runs the company whose models spent this month picking locks nobody knew were there.

He has every reason to say it, and he has said versions of it before. He also reads the eval results before we do.

Try it: Hugging Face turned its own bad week into a free guide on self-hosting an open model for cyber defense. It's the practical version of everything above, written by the team that just needed it. Send it to whoever would be running your incident.

Matt Van Horn has never written a line of code. He found out one of his products was 94.7% Go by reading his own GitHub page the day he shipped it. But here’s the thing, he’s actually a good engineer.

Last week he sat down with our cofounder Alex to discuss how he does it. He claims it all comes down to a text file.

Before he builds anything, he makes the agent write its own plan and save it. His reasoning is blunt: agents are lazy. They quit early or wander off. A plan on disk is the leash.

Here’s how you can do it too:

1) Pick work that's heavy on context, not code. A long meeting transcript plus the report someone told you to read. A competitor teardown. Three years of customer complaints. The plan file earns its keep when there's more material than a person wants to hold at once.

2) Ask for the plan before you ask for the work. Your first instruction isn't "do this." It's "write yourself a plan file, save it, and treat it as your source of truth." Then let it go. This one reorder is most of the value.

3) Hand over everything raw. Matt pastes in whole transcripts, whole books, screenshots. Do not summarize first. Summarizing is you doing the agent's job and throwing away the detail it would have used.

4) Make it plan the plan. Ask what it needs to read, in what order, before it starts. That extra step is what produced the book report nobody requested, and it's where the hour of unattended work actually comes from.

His closing argument is the one that should stick with anyone budgeting for this. As building gets trivially easy, the scarce thing isn't the building. It's knowing what's worth building. The full episode is free on YouTube.

None of this is required reading. But if you want to go further than your job demands, we've got you too.

The one take on the breach that isn't a recap — He walks the whole chain, then makes an argument nobody else is making: we've restricted the best models so heavily on security work that defenders can't use them on the attacks, while attackers face no such rule. The safety constraint may be making everyone less safe. Ten minutes, and the best thinking published on this all week. Simon Willison

The academic version of "don't let it read strange things" — A paper mapping the design patterns that keep agents safe from prompt injection, which is when instructions hidden inside content an agent reads get treated as orders. Dense. Also the clearest picture anywhere of the tradeoff between what an agent can do and what it can be talked into doing. arXiv

Sandboxing, for people who actually have to build it — Specific about isolation and permissions, written for production rather than for a slide. If somebody on your team is going to be handed this problem, this is the thing to hand them with it. Free. NVIDIA

An 87-year-old argument ended in one casual post — He dropped a counterexample to the Jacobian Conjecture on X, credited Claude, and it was short enough that other mathematicians checked it themselves within hours. The real story is that verification. This answer happened to be small enough for humans to confirm, and most won't be. The Conversation

Terence Tao on what AI does to mathematics, and one detail worth the click — The best living document on this, drawn from about 70 posts, 16 talks, and 6 essays, plus his brand-new lecture from Friday. The detail: the summary was drafted by Claude from his public writing, then corrected by Tao himself. The most careful mathematician alive used an AI to explain his views on AI, and says so at the top. Terence Tao