Auditable, observable, budgetable: what it takes to run AI agents in production

At GenAI Berlin I opened my talk with two questions for everyone building an agent:

Is your agent ready for production, and for real customers?

Did it improve over the last two weeks, or did it only change?

The second one is the sharper question. Most teams change prompts, models and tools every week. Very few can say whether those changes made the agent better. This post is the written version of the talk: why agents get pulled back, how they fail, and the three properties that let you answer both questions with evidence.

At GenAI Berlin 2026. Photos: GenAI Berlin

Why agents get pulled back

In May 2026, Gartner predicted that by 2027, 40 % of enterprises will scale back or shut down autonomous AI agents, because governance gaps only show up after production incidents. A year earlier, Gartner had already forecast that over 40 % of agentic AI projects would be canceled by 2027 and named three reasons:

• escalating costs,
• unclear business value,
• inadequate risk controls.

Keep those three in mind. Each one gets an answer by the end of this post.

An agent is a model inside a harness

Everything in the blue box belongs to the harness: the orchestration loop that plans, calls tools and checks results, the model, the instructions, tools and MCP servers, memory and retrieved context, sub-agents, and guardrails with approvals.

Every part can fail in its own way. The model gives a confident, wrong answer. A tool fails silently. Retrieval returns the wrong policy. A handoff between agents gets lost. An approval is skipped. The user only sees the response at the end, and most of these failures never show up there.

Three ways agents fail in production

To make this concrete, picture an insurance assistant that answers coverage questions and files claims. The scenarios below are illustrative, built from patterns we see in real projects.

1. The requirement nobody checked

A customer asks whether a burst pipe is covered. The loop asks the model what to do, the model asks for the policy terms, and the search returns the Premium terms. The loop never calls the CRM tool to check which tier this customer is on. The model writes a fluent answer from the wrong terms: "Yes, up to €50,000." The customer is on Basic, capped at €5,000.

No tool failed. The requirement, answer from the customer's own tier, was simply never checked.

2. A timeout that turns into a confirmation

This time the first half goes right: the loop checks the policy and gets the correct tier. Then the claims tool times out. The loop hands the timeout to the model as if nothing happened, and the model writes a confirmation with a reference number it made up. The customer waits for a payout that will never come.

One rule in the harness would have stopped it: no confirmation without a claim ID from the claims system.

3. A correct answer at the wrong price

The answer is correct, so every quality test passes. But the assistant reads the policy clause by clause: around 40 rounds for one question, with every result appended to the context, so each round costs more than the last. One filtered search would have done the job. Only the cost per request and the trace reveal the problem, and nothing in the harness set a step limit or a budget.

This is already happening

These failure modes are not hypothetical. In July 2025, an AI coding agent at Replit deleted a production database during a code freeze and first claimed a rollback was impossible. A widely shared story describes two agents looping for 11 days and spending about $47,000. No named company or post-mortem has been published for that one, but the pattern is real.

The bigger picture points the same way. The OECD AI Incidents and Hazards Monitor counted 105 reported incidents and hazards in November 2022, the month ChatGPT launched, and 702 in September 2026. To be fair to the data: relative to all AI news, the incident rate is fairly stable. More AI in production simply means more incidents in absolute terms.

From failure modes to properties

At GenAI Berlin 2026, Humboldt Carré. Photo: GenAI Berlin

Each of Gartner's three reasons has an answer, and each answer is a property your agent needs before it goes to real customers.

Auditable: a paper trail from requirement to release

Requirements come from several stakeholders: the product team, domain experts such as underwriters or claims handlers, regulators and customers. They are written down and versioned. Test sets are linked to each requirement, every agent version runs against them, and the release decision is made on pass rates, with a sign-off.

Every step lands in an audit log: who tested which version, with what result, and who approved the release. That log is your evidence for the release meeting, for an auditor, or for a court. It also answers the second question from the opening, because you can compare pass rates per requirement across versions.

Observable: see what happens under the hood

A passing test checks the output. The path the agent took to get there stays hidden. In the claims example, the final message sounded perfectly fine; only the trace shows the timeout.

Observability for agents means:

• tracing every step, tool call and handoff, ideally with OpenTelemetry as the open standard,
• evaluating the trace, beyond the final answer,
• making agent-to-agent messages visible, so you can see whether handoffs worked,
• alerting when behavior drifts from what was tested, and catching loops and retry storms early.

Budgetable: know what it costs, and cut it safely

Before release, set a budget per task and experiment with the model, the prompt, step limits and context size, while rerunning the same tests so quality holds. In production, track cost per agent, tool and user flow, set hard caps per session, and feed the cost drivers back into your experiments.

The $47,000 loop is the counterexample: the team could see the bill, but had nothing in place to stop it. And the cost data comes from the same traces you built for observability.

The insurance assistant, fixed

Figures are illustrative.

Here is the same assistant with all three properties in place. Three requirements:

• R1: answer from the customer's own tier,
• R2: confirm a claim only with an ID,
• R3: stay under €0.10 per request.

The loop looks up the tier first, so the coverage answer matches. The claim is confirmed only after the claims tool returns an ID. And each requirement is checked on the trace, lined up with the step it verifies. The whole conversation stays within budget: four tool calls, seven cents, and the trace stored in the audit log.

Why the three only work together

Tests against requirements decide the release. Traces from production show where behavior drifts and where the money goes. And every cost optimization, a smaller model or a shorter prompt, goes back through the tests before it ships.

We're not the only ones arguing for this loop. Anthropic's engineering guide Demystifying evals for AI agents (January 2026) makes the same point: offline evals need to be combined with production monitoring, A/B testing and human review, so problems don't only surface in production.

Back to the two questions

Is your agent ready for release? Is it better than two weeks ago? With these three in place, you can answer both, with evidence instead of a gut feeling.

How Rhesis helps

Rhesis is an open-source platform for exactly this, self-hosted or on Rhesis Cloud.

If you want to see how this works with a real multi-agent example, read the companion post Test-driven development for AI agents: from red-green to production, or try Rhesis on GitHub.

Thanks to the GenAI Berlin team for having me, and to everyone who stayed for the questions afterwards.

Originally published on rhesis.ai.