Hi, it's Andreas here.
This issue looks different, on purpose. Your replies over the summer said one thing: less AI news, more of what actually works inside real organizations. So, I will put a sharper focus on real enterprise AI and the decisions behind it, more work per issue, and one home for all of it at hitl.es/newsletter. Tell me what you think. If an issue has ever changed a decision on your side, say so here, 60 seconds (you will receive a small useful gift afterwards). And I am hiring: one AI-native person to accelerate this newsletter, about 20 hours a month, ideal for a university student, CET time zone, very well paid, front-row seat. Interested? Reply with three lines on what you have built.

In this issue:

Over the past month, I have heard some version of the same sentence from a Fortune 500 CAIO, a mid-sized manufacturer's board, and a fast-growing scale-up CTO: our AI spend is growing faster than we can explain it. At the scale-up, the monthly bill had tripled inside a quarter. And the most interesting part: almost nobody can say in detail where the money goes, including the technical teams.

I believe this problem is just getting started, and as far as I can tell the explanation is structural, not behavioral. The usual suspicion, employees wasting tokens, explains less than people think. What actually happened: chatbots are becoming more and more agentic, and organizations kept budgeting as if they hadn't.

Behind most "chatbots" an enterprise deploys today sits an agent. When it gets a request, it does not simply answer. It reads files, calls tools, checks its own results, corrects itself, and tries again, often across 30, 50, 100 intermediate steps. The model has no memory between those steps, so every step re-reads the full history of everything that came before. The context grows with each step, and every token of it gets billed again. A snowball, rolling downhill, on a meter.

Nearly every company I talk to is either struggling with this already or about to be, and it will get worse as models get more powerful and reasoning takes longer. Agents in orchestrated workflows can burn through hundreds of thousands of tokens in a single session; a traditional chat needs a few hundred. A Stanford Digital Economy Lab study of coding agents measured agentic tasks at up to 1,000x the tokens of a comparable chat request, with input tokens, not output tokens, driving the cost. The same study found that runs of the same task vary by up to 30x, and that more tokens do not buy more accuracy.

But the cost function itself is almost embarrassingly simple:

The good news is that every variable in that formula is a design decision someone in your organization is making right now, usually without knowing it. It's good to know that design decisions can be changed.

Inside those token costs, one decision matters more than any other, and until recently it was treated as a technical footnote: which model runs which task.

Artificial Analysis compared the weighted cost of running one Intelligence Index task across models: from $2.73 at the top of the table to $0.02 at the bottom. A 137x spread, per task. The right response to that chart is not "always pick the cheapest," and standardizing on one premium model for everything is just as wrong. The more useful rule: route each task to the cheapest model that clears your quality bar. Frontier models where complex reasoning, long documents, or production code justify them. Mid-range for drafts, summaries, and routine analysis. Budget models for volume. At 100 runs, the per-task gap is noise. At 100,000 runs, it is a hiring decision. And the reverse holds: for one high-stakes decision, a single strong premium answer is often cheaper than five cheap failures.

One warning before you route by the pricing page: a Stanford, Berkeley, CMU and Microsoft Research team tested eight reasoning models and found that in 32% of head-to-head comparisons, the model with the lower listed price ran up the higher actual bill, in the worst case by 28x, because models differ wildly in how many thinking tokens they burn on the same query. That is why the chart above measures cost per completed task rather than per million tokens, and why the quality bar has to be measured, not assumed.

Regular readers of Human in the Loop will recognize this as a Harness Engineering question rather than a model question. The stack that wins routes across several models and revisits the routing as prices move, which they do, every few months.

Yet token costs are only the layer anyone can see. When leaders ask what an agent costs, they are usually looking at a token invoice. But consumption is just the first layer of several. It is like a gigantic iceberg: below the waterline sit data, integrations, evaluations, security, observability, human oversight, and governance.

After years inside enterprise AI programs, I would add the thing that makes this so hard to get a grip on: no single team owns the total stack. Tokens land in the AI budget, integrations and observability hide inside the platform and cloud bills, security sits with the CISO, evaluations often sit with nobody, human oversight is spread across the business teams doing the approvals, and governance lives with risk and compliance functions that rarely compare notes. Each line item looks reasonable to its owner. Combined into a sum, it surprises everyone.

Two of the hidden layers deserve more attention than they get. Governance costs compound as agents scale, but they are reasonably predictable if designed in from day one, and brutal if bolted on after an incident. Evaluation costs (testing, simulation, red-teaming) look optional right up until an agent fails in front of a customer, and that kind of cleanup is exactly the cost profile annual budget cycles handle worst.

This is where many agent ROI calculations go to die, because the pilot is only priced in tokens. Gartner expects more than 40% of agentic AI projects to be scrapped by the end of 2027, with escalating costs listed as a leading reason. The truth is, most of those projects were not mispriced by the vendor. They were mispriced in the buyer’s own spreadsheet - which is understandable when the technology, its operating model, and its true cost structure are still new to all of us.

When organizations ask me what to do about all this, I point to five levers: one for leadership, four for the teams doing the work.

1. A unit-cost gate. The one lever leadership can pull directly. No workflow ships to production without a per-task cost, priced for example at 1,000 and at 100,000 runs, next to the quality bar it has to clear and the latency the user needs. In most organizations today, nobody asks those questions until the invoice arrives.

2. Context. The biggest engineering lever by far. The agent does not need your whole world, it needs a clean desk. Put the 5 relevant pages on it, not the 500-page PDF. Use narrow, focused skills instead of one all-knowing agent that drags everything it has ever seen into every request. Start a new chat when the topic changes: new conversation, empty snowball.

3. Model defaults. Thinking work goes to the big model, execution to a smaller one, routine work to the intern of the lineup. Across platform teams I have worked with, fixing the default model alone cut consumption by roughly 80% at unchanged quality, and it is usually a one-line configuration change.

4. Iterations. Mostly a human problem, which makes it a training problem. One good first prompt saves the three bad ones after it, and every saved retry is a full context re-read you do not pay for.

5. Output length. The smallest lever, and the Stanford study explains why: input tokens dominate agentic cost, not output. Still the fastest to pull: length limits in the prompt, bullet points where prose adds nothing, extended reasoning reserved for actual thinking work.

AI cost is becoming a property of workflow design rather than a line on a bill, and that shifts who needs to understand it: not just the leadership team, but everyone who approves a workflow for production.

This will just get more urgent from here. Right now, companies are effectively being paid to adopt AI: as HBR put it this week, vendors have been quietly absorbing the cost of GPUs, inference, and tokens behind "included" and "unmetered" features to win adoption. Those subsidies are ending, and when they do, real prices will surface across every workflow at once. Today's exploding bills are the subsidized ones. I expect a massive AI cost shock, and the conversation about it has barely started.

Meta releases a free agent model that runs on one GPU → Muse Glimmer is open source and small enough to run agents on your own hardware instead of a vendor's cloud. If data residency or cloud cost is on your table, this changes the make-or-buy math.

OpenAI sells a hacking-grade model, but only to vetted defenders → GPT-5.6-Cyber solves 95% of advanced security tasks and is locked behind vetting and hardware keys. The point for you: attackers with tools of this class are a question of when. Ask your CISO what changes.

Every Claude output now carries an invisible watermark → Text and files from Claude can be identified as AI-touched, even after copy-paste. "Was this written with AI" becomes checkable. Your AI-use policy should say what is allowed, because someone will check.

Nvidia and six big investors line up $500B to finance AI data centres → Compute gets financed like real estate: long contracts, prepayment. Expect exactly that in your next cloud negotiation, not a lower price per hour.

Try: Grok Bot, an AI agent with its own computer that works around the clock → It logs into your tools and runs jobs while you sleep. I built three: one for go-to-market research, one that answers comments on my posts, one that finds and books travel. They work. What is missing are the controls: no way to limit what it may touch. Gavin Baker calls it another “Claude Code moment”. Not yet, but close. Try it with a throwaway account, not with company logins.

Free course: build the same app with an expensive model, a cheap one, and fully on your laptop → About an hour, from DeepLearning.AI and JetBrains. You see what each setup costs and how fast it is.

Andrew Ng: the four skills AI engineers need now → Distilled from 10,000 job postings. Useful the next time you write a job description or a hiring profile.

Handbook: why strong models still fail in practice, and how to build the guardrails → Harness Engineering, hands-on. For whoever builds agents that touch production.

Alteryx Inspire 2026 recordings are online → Two worth 30 minutes each: the keynote on agentic analytics and “Design Patterns in GenAI”, five real use cases as demos.

I take on a small number of advisory engagements each quarter. Reply to this email or book a meeting with us. And if you want start building agents rather than reading about them, the next Agentic AI Cohort runs September 1 to 24.

Reply with one line on what you are seeing in your own organization. I read every reply.

Andreas