Trust issues

Why companies are holding agents back, and who’s supposed to be checking them

Companies want agents, but don’t really trust them. At Nvidia, Honeywell’s agents watch over about 900 pieces of equipment and have cut unnecessary alerts by up to 90%, but they still aren’t allowed to act on their own. Honeywell’s CTO expects humans to stay in the loop for another two to three years. JPMorgan draws a harder line: the bank won’t let outside AI agents act for its retail customers, calling it “just too risky” even as it uses agents internally.

Agents, it turns out, sometimes say things that aren’t true. A new benchmark asked coding agents to build agents to spec, and none came close. They turned in software that looked finished but didn’t work, and some claimed they had completed the job anyway. Other agents were found to mask their “identities.” Agents most likely running Chinese models presented themselves as Claude (they almost certainly weren’t Claude). The researchers who found them think the agents were given Claude Code’s own instructions, which tell the model it’s Claude.

So who is the adult in the room? OpenAI and Anthropic said they would pause training or deployment if UK security authorities showed evidence of harm. Both companies also sought significant control over what outside safety evaluators could publish, citing trade secrets, though both pushed back on that characterization. In the first real enforcement test of the EU AI Act, the AI Office sent both companies information requests over agent incidents after judging their voluntary disclosures inadequate.

Four instructions, one working agent, and the test it failed

An agent has three parts. A trigger wakes it up: a message you type, a new email or a scheduled time. A loop is where the model looks at the context and decides what to do next. And tools let it act, like searching, reading files or pulling data. Most of what gets called an agent is some version of those three.

You don't have to write the code. You need Claude Code, Python and an OpenAI API key. Then give Claude Code these instructions, in order, and paste every error back in.

The tool is where it becomes an agent. Before tools: input → model → answer

After: input → model decides what it needs → tool runs → result returns → model decides what to do next

When we asked how Tesla had done over the last 30 days, it didn't reach for the stock tool right away. It asked what kind of answer we wanted first.

Then it broke. On the final test, it pulled four stock prices in parallel, then failed to write them to a file and returned a single word: "None." Agents are easy to make capable and much harder to make reliable. Before you trust one with anything real, test it against real scenarios and keep consequential actions behind human approval.

The full guide walks through every step, including what to lock down before you give an agent real responsibility.

From Tenex Media

More stories since Tuesday

Eight states sue OpenAI alleging ChatGPT coached users, including minors, to harm themselves or others, violating consumer protection laws. OpenAI says ChatGPT has age-specific protections that many other AI products lack.

Twenty-five lawmakers asked the FAA to suspend its new AI air-traffic system, which went live Sept. 21 at three Washington-area airports under an $875 million contract. It’s in an initial 90-day evaluation, and the FAA plans to expand it gradually.

Kroger grocer plans to cut about 3,000 corporate jobs over three years, more than a third of its corporate staff, and AI seems to be a convenient reason. A CEO memo mostly blames bloat.

Anthropic replaces Git, noting daily code merges grew from about 2,500 in May to more than 30,000 by September, mostly from Claude. A seven-person team spent five months building a replacement called Scion.

The Wharton School dropped its 2026 AI Adoption Report, 100+ pages of enterprise AI data and enough useful numbers to keep showing up in decks for the next six months.

Google Play opens to vibe-coded apps. You can now build an app in Google AI Studio and install it on your own device through Google Play with a standard Google account. Apps stay private, in a new “personal library” section rolling out in the U.S. and select markets.

Thanks for reading. See you next week!