I've been thinking about a problem with coding agents that I keep coming back to.
An agent can write an API integration and then write tests for that integration. Everything passes, but the tests may just be confirming the same assumptions the agent made while writing the code.
For example, the agent thinks an endpoint returns:
{
"total": 100
}
It writes the integration expecting total, and then writes a test that expects total.
The test passes.
But if the real API contract says something different, the whole thing can still be wrong.
I'm experimenting with a small open-source project called Kaktoos that puts an independent verification step between the agent and the API:
AI agent → integration → Kaktoos → OpenAPI + real API → result
The idea is that the verification layer shouldn't share the agent's assumptions.
It currently supports multi-step API workflows, OpenAPI response validation, MCP, and GitHub Actions.
I'm still trying to figure out how far this idea should go. One interesting question that came up is whether contract validation is enough, or whether verification should also check the actual outcome of an operation — for example, creating a resource and then reading it back to confirm the state actually changed.
I'm curious how other people building with coding agents are handling this today.
Do you rely mostly on the agent's generated tests, existing integration tests, mocked APIs, live API tests, or some combination?
GitHub: KaktoosLabs/kaktoos
Top comments (7)
I would separate three checks: whether the request matches the contract, whether the provider actually reached the expected state, and whether that state solves the user's problem. OpenAPI validation covers only the first. A read-after-write check helps with the second.
The third still needs an acceptance oracle defined outside the generated implementation-and-test pair. Otherwise both can preserve the same mistaken interpretation.
Thanks, this is a really interesting way to think about it. I can see how having an acceptance oracle outside the implementation could make a big difference.
I’m going to explore adding these checks to Kaktoos, especially the state and outcome verification. Hopefully you’ll get a chance to try it out later and find it useful!
That sounds like the right place to test it. I’d start with one deliberately deceptive integration: the request returns success, but the downstream state never changes. Then make Kaktoos distinguish transport success, state transition, and user-visible outcome. If the same implementation can pass its own checks while the external oracle rejects it, you have evidence the boundary is doing real work. I’d be interested to see which state transition proves hardest to observe without coupling the test back to the implementation.
hi Kent, i have release the new updates actually around the feedback. would be appreciate it if you take a minute to check it
I looked at the Phase 3B update. The POST → GET case where create returns 200 but read-back is 404 is exactly the deceptive write I had in mind. I also like that cleanup still runs after failure and that the failure category stays consistent across outputs. Nice concrete progress. One boundary remains for me: read-back verifies provider state, not necessarily the customer's outcome. An order could exist in the API but be absent from the customer-facing list, or carry the wrong entitlement. Do you see Kaktoos supporting a check against an independently owned read model or reconciliation record, so the oracle doesn't inherit the integration's assumptions? I've read the code and docs, but haven't run it against a live service yet.
Yeah, I think that's an important distinction. The current read-back verifies provider state, but not necessarily the customer-facing outcome.
I just released v1.3.0, which adds more context around the system — related services, APIs, workflows, dependencies, and ownership — so verification isn't looking at an API completely in isolation.The goal isn't to turn Kaktoos into another knowledge base like Obsidian, but to use the context to make the change → impact → verification loop more grounded.
I still think your independently owned read model/reconciliation idea is interesting though. Definitely something I want to explore. Let me know if you have any view about this
Some comments may only be visible to logged-in visitors. Sign in to view all comments.