A question for engineers building AI agents that actually change things.
Imagine this fictional workflow:
- An agent is approved to update an ERP record.
- The role, task scope, and review binding pass preflight.
- The tool definition changes before dispatch.
- The gateway forwards the operation, but the client times out.
- Another agent takes over and considers retrying.
What evidence would you need before trusting the result?
I'm exploring four possible ingredients for a minimal evidence chain:
- The exact approved task/action digest
- The authenticated executor and effective tool version
- The logical operation ID and retry identity
- A fresh, authoritative readback of the actual result
But every extra receipt has a cost: storage, implementation, maintenance, and operational complexity.
I maintain Agent Role Contracts (ARC), an open-source project that checks supplied role, scope, review, and task/action declarations offline. Runtime enforcement and independent verification of external effects are separate responsibilities.
We're investigating these boundaries through a research pilot, with the real-world validation stages explicitly tracked.
My question: What's the smallest evidence chain you would trust in production?
Have you solved this with database logs, idempotency keys, event sourcing, workflow engines, or something simpler?
A real example would be especially useful:
system | side effect | failure | trusted evidence | retry rule | cost
I'm collecting concrete engineering perspectives in this GitHub discussion as well.
Top comments (18)
The timeout-after-dispatch step is the killer case — preflight passed, the tool changed, the client timed out, and now nobody knows if the ERP record moved. Of your four ingredients, the fresh authoritative readback is the only one that actually answers the question; the rest prove intent, not effect. In your pilot, are you treating the readback as mandatory for every mutating call, or only for the high-stakes ones?
Good question. I'd make the strength of verification proportional to the risk, while requiring credible effect evidence before declaring a mutating workflow DONE.
For low-risk operations, a durable receipt from the authoritative destination may be sufficient. For high-impact changes, timeouts, or agent handoffs, I'd require a fresh, independently obtained readback tied to the logical operation ID.
If the result cannot be established, the status remains UNKNOWN, and an automatic retry should not proceed on that uncertainty alone.
ARC checks the supplied declarations. Runtime enforcement and durable effect verification are separate layers, and our joint ERP pilot has not yet established an end-to-end result.
I'm particularly interested in the latency and operational cost of that final readback. Have you encountered a case where its cost materially changed the design?
Risk-proportional verification is the right shape — a durable receipt for low-risk calls, a fresh readback tied to the operation ID where the stakes are real. On the readback cost: the read itself is rarely the expensive part; what changes designs is when the destination has no clean read API and the 'readback' becomes a second workflow. Is that the case in your ERP pilot — a system where verifying the effect costs nearly as much as the effect itself?
Exactly — that's the distinction we're trying to make measurable.
To clarify, the ERP workflow in the article is a fictional scenario. Our ARC research is examining these verification boundaries, and we haven't claimed a completed production ERP pilot.
Our working hypothesis is risk-proportional verification:
Your point about readback becoming a second workflow is especially important. In legacy ERP systems, verification complexity may become a deciding factor in whether an agent should execute an operation autonomously.
Have you encountered a practical way to keep verification costs manageable when the target system only exposes a UI or batch exports?
Thanks for clarifying — the risk-proportional framing is a good answer to my question. On the UI-or-batch-only case, my honest take is that the cheapest verification is often not verifying at all: if the agent can't read back what it changed, that operation probably shouldn't be autonomous yet. Batch exports at least give you a reconciliation point, even if it's a day late; a UI-only target gives you screenshots and hope.
That's a useful distinction. I think you're right: if an agent cannot establish what changed, the right optimization may be to reduce autonomy rather than weaken verification.
I'd distinguish three cases:
This suggests a broader principle for our ARC research: autonomy should be constrained by what the system can actually verify, not just what the agent is authorized to attempt.
One question: would you consider delayed batch reconciliation sufficient for low-risk autonomous writes, provided retries remain blocked until reconciliation completes?
Yes — for low-risk writes I'd take that deal, with one condition: the blast radius has to stay bounded while the result is UNKNOWN. Blocked retries plus a clear unknown state is the key part; the failure mode you're avoiding is the agent retrying its way into a duplicate. The moment the write touches money or external state, though, I'd want the human back in the loop even at low risk — some unknowns aren't worth batching away.
@suirindo This scenario exposes two subtle traps that standard idempotency keys and generic entity readbacks fail to solve:
GET /record/42). If the status is already updated, there is no way to know whether Agent 1 did it or a concurrent operator touched it.In regulated biobanking LIMS and ERP systems (where duplicate specimen thawing or relocation is catastrophic), we solve this with a tripartite invariant:
UNKNOWN, neverFAILED)correlation_id,caller_role,tool_sha256,applied_at) commit within the exact same physical database transactioncorrelation_id. IfFOUND, it returns the committed state; ifNOT_FOUND, it evaluates the live rowversion before authorising a single idempotent re-submissionTwo architectural rules close the remaining loopholes:
HMAC(task_digest | tool_definition_sha256 | entity_id). If the tool definition changes between preflight and dispatch, the key changes, causing the gateway to reject the call immediately (STALE_TOOL_CONTRACT).correlation_id, not just the entity state. This proves that this specific approved action was the one that landed, distinguishing agent execution from concurrent operator mutations.Tom, this is a valuable production example. Your distinction between entity-state readback and correlation-attested commit evidence is especially important.
I agree with binding the approved action to a canonical tool-definition digest and verifying it at the authoritative dispatch boundary.
Two details seem worth testing explicitly:
Stale contract rejection: Changing the HMAC-derived idempotency key alone would not reject a stale contract. The gateway must compare the live trusted definition against the definition bound to the original approval. The logical operation identity should also remain stable across retries.
Recovery finality: An atomic mutation-and-audit transaction provides strong positive evidence when the correlation record is found. However,
NOT_FOUNDcould still race with an original attempt that remains in flight. A rowversion check alone may not resolve that ambiguity.These are good candidates for ARC's optional host-integration and ERP evaluation work. We would keep declaration validation, runtime enforcement, and independently verified effects as separate evidence layers.
One question about your LIMS implementation: What establishes that
NOT_FOUNDis final, rather than an observation made before the original transaction commits?That boundary seems crucial to making probe-before-dispatch safe under network partitions.
@suirindo That cuts straight to the core challenge of distributed systems under network partitions: an unfenced
NOT_FOUNDis only an observation of transient visibility, not a proof of finality.In high-assurance biobanking and financial LIMS, a passive
NOT_FOUNDis never treated as final on its own. Finality is established through three complementary transactional boundaries:Lock-Coordinated Probing (Eliminating In-Flight Ambiguity)
A recovery probe does not execute a raw read. It queries the journal under a transactional coordination lock (such as an application lock or exclusive row lock on the
correlation_id). If the primary attempt is still in-flight, the probe is forced to wait until that transaction either commits or rolls back. Once the lock yields,NOT_FOUNDis guaranteed final because the primary attempt has definitively terminated.Atomic Unique Fencing (The Zero-Trust Backstop)
Even in an adversarial partition where gateway restarts sever the connection, the database audit journal enforces a strict physical
UNIQUE CONSTRAINTon thecorrelation_id. Any subsequent retry by a handover agent must inherit that identical correlation token. Even if an agent raced ahead, the database engine physically guarantees that two executions can never both commit; the second will instantly abort with a constraint violation.Deterministic Connection Expiry
The database server enforces a hard query timeout that is strictly shorter than the handover agent's recovery delay window. A recovery probe is barred from declaring finality until that connection-lifetime threshold has elapsed, ensuring orphaned transactions are severed before recovery dispatches.
While these three invariants are straightforward in theory, executing lock-coordinated probing under high-concurrency partitions without deadlock spirals is non-trivial.
Since you mentioned tracking these boundaries in your active research pilot for ARC, this is the exact transactional runtime engineered at Biomed Systems Limited for regulated environments. If your pilot partners need a hardened, production-tested host-verification engine rather than implementing the relational fencing from scratch, my advisory and direct contact channels are in my bio.
Thanks, Tom. This is a valuable explanation, particularly the distinction between observing
NOT_FOUNDand establishing transactional finality.I think your approach gives us three important assumptions to verify:
Lock coverage: Both the original dispatch and recovery probe must participate in the same durable coordination mechanism before mutation, including when no journal row exists. Otherwise, a late primary attempt could still race with recovery.
Atomic effects: The unique correlation constraint must protect the actual business mutation within the same transaction. A unique audit record alone cannot prevent duplicate external side effects.
Verified termination: A query or connection timeout must correspond to a demonstrably terminated or rolled-back transaction before recovery can safely proceed.
These are exactly the kinds of host-side guarantees we want to distinguish from ARC's declaration-level checks.
I'd be interested in testing one concrete failure scenario from your LIMS architecture: an in-flight primary operation, gateway disconnection, recovery probe, and competing handover retry.
Would you be open to sharing a sanitized execution trace or minimal reproducible test, including the observable receipts and recovery rule?
That could provide valuable independent evidence for ARC's host-integration research. We'd be glad to acknowledge Biomed Systems' contribution in any public evaluation, with your agreement.
Thanks again for bringing real production engineering experience into this discussion.
@suirindo Those three assumptions capture why host-level relational mechanics cannot be approximated in user space:
Lock coverage on cold keys: When no row exists in
dbo.SYS_DispatchJournal, row-level locks have no physical anchor. In my architecture, the stored procedure opens by acquiring a transaction-owned application mutex (sp_getapplock @Resource = @CorrelationId, @LockMode = 'Exclusive', @LockOwner = 'Transaction') before checking existence. Both primary dispatch and recovery probes must acquire that identical mutex key, serialising the race even before the first row is inserted.Atomic effects within a single transaction: The correlation journal insert and the business mutation (
UPDATE/INSERTon specimen or inventory state) execute inside the same physical transaction underSET XACT_ABORT ON. If a duplicate retry hits theUNIQUEconstraint or a lock conflict, the database engine aborts and rolls back the entire unit of work atomically. Separate audit writes outside the business transaction cannot prevent duplicate side effects.Verified termination before recovery: A client socket timeout (
ECONNRESET) never proves the database server has finished rolling back. Rather than guessing from elapsed wall-clock time alone, the recovery probe tests the correlation mutex with an immediate lock timeout (SET LOCK_TIMEOUT 0). If the orphaned session is still active or mid-rollback, the lock acquisition fails immediately (Error 1222), forcing the handover worker to holdUNKNOWNuntil the engine releases the transaction lock.Regarding a reproducible test for your evaluation: the standalone Node.js verification matrix (
v1.1.0on my profile) is my complete open reference contribution. Production LIMS failure traces, relational fencing harnesses, and ERP recovery adapters are commercial IP delivered through Biomed Systems Limited under paid enterprise pilot or advisory engagements.If Netsujo or a commercial partner has budget allocated for a formal host-integration pilot, my direct commercial channels are listed in my profile.
Thanks, Tom. This is a precise and useful explanation.
The key insight for us is that recovery finality depends on host-level transactional coordination. Both the original dispatch and the recovery probe must participate in the same locking protocol, while the business mutation and correlation journal commit atomically.
This gives us a useful distinction for ARC's evaluation work: validating an agent's authority, controlling execution, and establishing independently verifiable evidence of the resulting effect are separate assurance layers.
We'll examine your public v1.1.0 verification matrix as a reference. I also appreciate the clarification about your commercial IP boundaries and would be glad to keep the dialogue open as our integration research progresses.
One small implementation question: SQL Server documents
sp_getapplocktimeout as return code-1, while error 1222 can arise from statement-level lock timeouts. Does your recovery probe handle both paths explicitly?That distinction could make a valuable reproducible failure case.
Thanks again for sharing the production architecture perspective.
@suirindo Yes, robust recovery probes must branch on both: driver-level Error 1222 (statement lock timeout) and the stored procedure return code -1 (applock timeout), wrapping both into a unified
TRANSIENT_LOCK_TIMEOUTrecovery state.Glad the high-level transactional principles were helpful for ARC's conceptual boundaries; looking forward to hearing how your integration research progresses if you decide to explore a formal collaboration.
Thanks, Tom. That distinction is very useful.
I'd treat Error 1222 and the
sp_getapplockreturn code-1as separate detection paths, normalized intoTRANSIENT_LOCK_TIMEOUTat the recovery-probe layer. Crucially, the underlying business operation would remainUNKNOWNuntil authoritative, correlation-bound evidence establishes its outcome. A probe timeout alone should never authorize a retry.For ARC, this is a valuable host-integration test case, separate from declaration validation.
A useful next step would be a small synthetic failure-injection matrix covering both timeout paths, reconciliation, and retry authorization — without requiring proprietary LIMS implementation details.
I'd be interested in comparing such results with your published work and exploring a more formal collaboration if there's a concrete technical fit.
Thanks again for sharing these details.
The timeout-after-dispatch case is the one that convinces me too — a plain readback can't tell you whether your agent or someone else made the change. I'd want the correlation-attested readback Tom describes, plus a human approval step before any retry on a high-stakes record, so an UNKNOWN stays UNKNOWN until someone has actually checked it.
Thanks, Elarigent. I think you're identifying two separate controls: evidence that this specific approved operation caused the change, and authorization to retry after an uncertain outcome.
A correlation-attested readback is meaningful when the correlation record is authoritative and atomically bound to the business mutation. After a timeout,
NOT_FOUNDalone still doesn't prove the original operation won't commit later.For high-stakes records, I'd keep the outcome
UNKNOWN, block automatic retries, and require host-side reconciliation plus explicit human authorization before redispatch. Human approval should govern the retry decision, not substitute for evidence.This is an important failure case for our ARC-related host-integration research. ARC's current declaration checks remain separate from ERP commit verification.
Have you seen teams make this distinction explicit in their operator workflows?
Tom's tripartite invariant is the right spine. Two traps I'd add from the retry path.
First, the readback has to come from the authority, not the nearest copy. A handover agent that checks a read replica - or a cache, or a search index - can see a stale "not applied", retry a write that already committed on the primary, and manufacture the duplicate it was trying to prevent. Replication lag turns ingredient 4 into a false negative at exactly the moment it matters. Same rule as Tom's audit ledger: the evidence read must live in the same consistency domain as the write.
Second, on the smallest-chain question: ingredients 1-3 collapse into one artifact if you write the intent before dispatch. One row - task digest, executor, tool sha256, operation id - committed ahead of the mutation, then the mutation and its commit marker in the same transaction. Four receipts become two writes, and the retry rule reduces to one lookup: did this exact intent commit? Everything else can be reconstructed at incident review, which is when you have time to read it.