Every agent I have ever shipped was qualified the same way: someone watched it work once, nodded, and called it production-ready. I did this for years. I trusted my own demos.
So I built the thing that stops me. It's called HivePlane — an open-source control plane where an agent cannot touch a production context until it has passed a reproducible benchmark and holds a signed attestation. Last week I ran its first full field test against real agents: 10 scenarios, 20 acceptance criteria, all pass. The most interesting moments were the refusals.
If you run more than one agent in anything resembling production, this story is for you. And if your agents already benchmark-gate their promotions, tell me where — I looked for that tool and couldn't find it, so I'd genuinely like to hear about yours.
The sentence that started it
I kept writing the same note while building this thing:
An agent is production-ready because someone watched a demo.
That sentence is true at every company I've worked for, and it's true for a worse reason than laziness: there is nothing to check. Agent frameworks solve orchestration inside one workflow. Nothing solves the fleet-level questions — who owns this agent, what may it spend, which tools may it call, and the one nobody answers: has it proven itself?
SWE-bench gave coding agents a reproducible benchmark and clear pass/fail, and coding agents got dramatically better. Production agent fleets have no equivalent. Teams swap prompts, change models, and ship to production with zero benchmark evidence — then act surprised when a regression reaches a customer.
The loop, and the ladder inside it
The control plane enforces one loop:
register → certify → gate → run → intervene → deliver → observe
The part I care about is the ladder. Every workload carries a certification status, and admission is enforced against it:
| Status | Meaning | Where it can run |
|---|---|---|
uncertified |
Registered, never benchmarked | Sandbox only |
provisional |
Passed the staging threshold (0.80) | Staging |
certified |
Passed the production threshold (0.90) | Production |
quarantined |
Failed re-certification or drifted | Runs blocked |
The ladder is the product. Budgets, policies, and dashboards exist to make the ladder real.
tip: Production certification is not the staging threshold with a bigger number. It's a separate benchmark run, and a manifest change — prompt, model, tools — invalidates the old certification. There is no silent path back into production.
What the field test proved
The receipts are committed in the field test report: 10 scenarios against the live Docker stack, 10/10 pass, all 20 acceptance criteria holding. The subjects were real agents — a raw-Python support agent and a LangGraph judge graph — plus four deliberately bad fixtures I'll cover in article 3.
Certification is not a simulation, which is the part I'd underline twice. The benchmark executes every corpus task as a real run — through the runtime adapter, the policy boundary, and (where used) the governed model seam:
| Workload | Context | Status | Pass rate | p95 latency |
|---|---|---|---|---|
| support-agent | staging | provisional |
1.00 (6/6) | 92 ms |
| support-agent | production | certified |
1.00 (6/6) | 67 ms |
| eval-judge | staging | provisional |
1.00 (4/4) | 215 ms |
| eval-judge | production | certified |
1.00 (4/4) | 126 ms |
Every certification produces a signed Ed25519 attestation bound to the exact model identity, verified on every read. The same sweep refused an uncertified agent with a 403, blocked a model swap with a 403, quarantined an agent that looked fine and wasn't, and killed a run the moment it went over budget.
The container layer separately went 25/25 — image build, API contract, the control loop, restart durability, the UI — because a control plane that only works on my laptop is not a control plane.
Why "Kubernetes for agents" is the honest analogy
Kubernetes is not a container runtime; it operates many of them behind one contract. This is not an agent framework; it operates many agents behind one contract — the workload manifest: owner and team, runtime adapter, allowed tools, model identity, budgets, sandbox and egress rules, certification corpus and thresholds, fan-out destinations.
Change anything in that manifest and the agent re-certifies before it can touch production again. That last sentence is the whole thesis, and the rest of this series is what it cost to make it true.
What I learned building it
The refusals are the product. Any platform can start runs. The gate that says 403: certification status 'uncertified' is insufficient for production; requires 'certified' — named, attributed, actionable — is what makes the platform trustworthy. A refusal an operator can act on is a workflow; "Forbidden" is a dead end.
Certification is a security control, not a quality metric. The moment production admission depends on a signed attestation, a whole attack class becomes blocked and auditable: swap the model, edit the manifest, quietly regress the agent. Article 3 is the four ways my own gate said no.
The operator surface has to be fast or it won't be used. Inspect-plus-stop measured at 0.04 seconds; scaffolding a new project at 0.24 seconds. Slow governance tools get bypassed, and a bypassed control plane is a expensive dashboard.
What it doesn't do yet
- v0.1.0 certifies two adapters: raw Python workers and LangGraph. The adapter contract is the seam; broader framework coverage is deliberately deferred.
- The drift detector ships next. Today, re-certification is scheduled and change-triggered — it catches what changes, not what fades.
- One model identity was validated this cycle; the cloud-profile run with real prices is the next field test.
References
- Field test report (v0.1.0) — source for every number above
- Docker test report — the 25/25 container layer
- Security audit · Release notes · Changelog
Next in this series: why my three "real" agents failed the field test on day one, and what I certified instead.
What's the last agent you shipped on demo evidence alone — and what did it cost you when it mattered?
Top comments (4)
The sentence doing the work is that a manifest change invalidates the old certification. Most gates I have seen treat promotion as a one-time event and then let the thing underneath drift, which means the attestation is a statement about a version that no longer exists. Binding it to model identity and re-verifying on read is the version that survives a model swap someone made on a Friday.
Where I would push is what the manifest does not contain. Owner, adapter, tools, model - all yours. The upstream services the agent calls are not, and they change without telling you. Same prompt, same model, same tool list, a payment API that started returning a new field, and the certification is still green because nothing inside it moved. That is the failure I keep writing about: the check did not fail, it stopped being about the thing it was checking.
Which makes 10/10 the number I would be least comfortable with, and you are already halfway to my point by saying the refusals were the interesting part. A gate that has never blocked a promotion someone actually wanted is the same evidence as a gate that cannot block one. Your four bad fixtures are the right instrument. What I would add is a block rate on real traffic tracked over time, with zero treated as an alarm rather than a pass.
One question, because it is the part I have not solved either. When re-certification fails and the workload quarantines, who is allowed to override it, and does the override land in the attestation chain? Every gate I have watched eventually grows a person who can say yes, and that person is usually where the audit trail stops.
The sentence that lands is the same one that worried me for years: an agent is "production-ready" because someone watched a demo.
A certification ladder (uncertified → provisional → certified, with manifest changes invalidating the attestation) is the right shape. Where I'd push next is what the corpus is allowed to omit.
If the benchmark only scores calm-sky happy paths, the attestation is a statement about a sky that never storms. I'd want at least one arm where:
Upstream drift (arun's point) is the cousin of that: same prompt, same model, same tool list, a dependency that changed shape — certification stays green because nothing inside the manifest moved. Treating a long streak of zero real blocks as an alarm, not a pass, is how you keep the gate honest.
Refusals with named, actionable reasons > "Forbidden." That's the product.
Agent demos can be so misleading! They always work perfectly in the demo but break in real life. Great that you built a gate to catch this.
For anyone testing AI agents, I've been using jzstoken for my API keys. Makes it easy to test different models without signing up for each one separately. They give you $5 free to start, so you can run real-world tests without any upfront cost.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.