DEV Community

Cover image for AI Agents Are Making an Old Distributed Systems Bug More Expensive
AdmilsonCossa
AdmilsonCossa

Posted on Edited on

AI Agents Are Making an Old Distributed Systems Bug More Expensive

The dual-write problem existed long before LLMs. Now a duplicate event can trigger fraud investigations, payment holds, refunds, or financial execution.


At 11:57 PM on Black Friday, a global commerce platform approves a €2.4 million payout to a merchant.

The ledger commits the transaction. The application publishes PayoutApproved. Then the request times out.

Nothing is necessarily down. The database is healthy. The broker may also be healthy. The network recovered milliseconds later. Every dashboard is still green.

Yet the system has lost something more dangerous than availability.

It has lost certainty.

Sequence diagram showing a payout database commit followed by an event publication whose acknowledgement is lost, leaving the producer uncertain whether to retry.

Did the broker receive the event?

If the producer does nothing, the risk system may never analyze the payout. If the producer retries — and the first publication actually succeeded — the event may be delivered twice. If downstream execution is not idempotent, the same business operation may be applied twice.

A few milliseconds of uncertainty can become a multimillion-euro question:

Did the operation fail, or did we merely fail to observe that it succeeded?

This is not fundamentally a Kafka problem, a NATS problem, or an AI problem. It is a distributed systems problem.

Modern AI agents make it more expensive because event consumers are becoming more capable. A duplicate event no longer has to mean a duplicated projection or a repeated log line. It may wake an agent that assesses fraud, recommends a payment hold, launches an investigation, requests a refund, or calls a tool connected to an external financial system.

The delivery semantics did not get worse. What sits behind the event became more powerful.

The architecture needs answers to six questions:

  • Where does the event become authoritative?
  • How is publication recovered after failure?
  • How do consumers tolerate redelivery?
  • How is ordering preserved where the business requires it?
  • How does each irreversible effect receive a stable identity, survive executor crashes, and remain reconcilable when its outcome is unknown?
  • How does a probabilistic assessment become one authoritative, deterministic decision?

1. Your Broker Does Not Solve the Dual-Write Problem

A service that approves a payout typically performs two operations that conceptually belong together but live in two separate transactional systems: the database and the broker.

Flowchart showing the database update and event publication in different transaction boundaries.

There is no automatic atomic transaction spanning both systems, which creates three distinct failure states:

Failure Consequence
Database commits, event publish fails Business state exists, downstream systems never learn about it
Event published, database transaction rolls back Consumers observe something that never became true
Broker accepts event, ACK is lost, producer retries Possible duplicate delivery

This is the dual-write problem. A broker gives you decoupling, buffering, durability, fan-out, replay, and failure isolation. What it does not give you is atomicity between an event and the database state that caused that event to exist. That distinction is the foundation for everything that follows.


2. Broker Guarantees Stop at Their Transaction Boundary

The useful question is not “which broker is best?” It is:

Which state transitions can this product make atomic, and where does that atomicity stop?

Kafka and Redpanda can provide transactional guarantees across supported broker operations. Kafka Streams, for example, can atomically combine consumed offsets, state-store updates, and produced records within the Kafka transaction boundary.

RabbitMQ provides durable queues, publisher confirms, acknowledgements, quorum queues, and routing mechanisms. NATS JetStream provides durable streams, acknowledgements, redelivery, and time-bounded publication deduplication.

These are valuable guarantees, but they belong to different boundaries:

Boundary Guarantee that may be available
Broker record → broker record Broker transaction or atomic stream-processing operation
Producer → broker Confirmed or acknowledged publication
Broker → consumer Durable delivery with acknowledgement and redelivery
Application database → broker Not atomic without an Outbox or equivalent mechanism
Broker → external provider Not atomic
Internal ledger → bank transfer Not atomic without a provider-level uniqueness and reconciliation contract

A broker transaction cannot atomically commit an unrelated Postgres transaction, a payment-provider request, or a bank transfer.

Choose the broker from the workload. Design correctness from the transaction boundary.


3. AI Agents Don't Create the Problem — They Increase Its Blast Radius

The dual-write problem predates LLMs by decades. Retries, duplicate messages, lost acknowledgements, and reordered events are established distributed systems failure modes. What's changed is what a consumer can now do with a duplicate.

Low-risk event path from PayoutApproved through a projection consumer to a dashboard update.

Higher-risk event path from PayoutApproved through an AI risk agent, fraud recommendation, policy decision, and payment action.

If the old projection consumer processes a duplicate twice, and the projection is idempotent, the impact may be negligible. If the same duplicate now traverses an agentic path, it can trigger a second fraud investigation, a second account block, or a second refund instruction.

That doesn't make AI agents inherently unsafe. It means the architecture around them must assume messages arrive more than once, agents can be wrong, tools can time out, models can change, and external systems can return ambiguous outcomes.

Once an agent participates, the architecture must distinguish three identities:

  • eventId — the immutable identity of a fact that happened;
  • decisionRequestId — the identity of one request for an assessment or decision;
  • operationId — the identity of one intended external business effect.

These identifiers are related through causation, but they are not interchangeable. One event may legitimately trigger multiple decisions, and one approved decision may produce multiple external operations. Reusing eventId for every boundary can suppress legitimate work or accidentally collapse distinct effects into one.

An AI agent should never move money simply because it generated a plausible answer. The safer boundary separates reasoning from execution:

Safe execution boundary separating an AI assessment, deterministic policy decision, durable external operation, and payment execution.

AI recommends. Policies decide. Systems execute.

The agent analyzes, classifies, explains, and recommends. A deterministic service evaluates limits, permissions, segregation of duties, and compliance. Only then does an idempotent executor perform the external effect.

A retry of the same decisionRequestId should return the already-recorded assessment. A reevaluation caused by new evidence, a new aggregate version, or an explicit review request receives a new identity. Idempotency here does not require a probabilistic model to generate identical tokens; it prevents a transport retry from creating a second authoritative decision.


4. CDC Is More Than a Legacy Escape Hatch

Enterprise systems rarely start clean. Banks, insurers, and telecoms often depend on ERPs and settlement engines built years before event-driven architecture became mainstream. Rewriting them to emit domain events may be too risky.

Change Data Capture — commonly Debezium — observes transactional database logs and captures committed changes without touching the legacy application.

CDC flow from a legacy database through log capture and a translation layer into stable domain events.

The trap: a row mutation is not automatically a domain event. STATUS = 'A' might mean approved, awaiting review, or an internal transition with no meaning outside the legacy system. If downstream services must understand physical tables and cryptic status codes, the database schema has silently become a public contract — and a fragile one.

A translation layer protects the rest of the organization from the legacy system's physical representation. CDC is also strategic beyond migration: it can feed search indexes, warehouses, read models — and most importantly, it can capture an Outbox table, transporting events the application deliberately designed rather than technical noise.

CDC also inherits the ordering and transaction semantics of the source log. A translation layer must preserve transaction identity and commit order where the downstream business invariant depends on them; row-level change order alone is not automatically domain-event order.


5. Transactional Outbox Fixes the Birth of the Event

A new payment service knows two facts at the same instant: the payout changed state, and an event describing that change must eventually be published. Instead of two independent writes, it records both facts in one local transaction.

BEGIN;

WITH advanced_payout AS (
    UPDATE payouts
    SET status = 'APPROVED',
        version = version + 1
    WHERE id = 'pay_7f9q'
      AND version = 16
    RETURNING id, version
)
INSERT INTO outbox (
    event_id,
    aggregate_id,
    aggregate_version,
    event_type,
    payload,
    occurred_at
)
SELECT
    'evt_01K...',
    id,
    version,
    'PayoutApproved',
    '{"payoutId":"pay_7f9q"}',
    CURRENT_TIMESTAMP
FROM advanced_payout
RETURNING aggregate_version;

-- The application must require exactly one returned row.
-- Zero rows means that the expected version was stale.

COMMIT;
Enter fullscreen mode Exit fullscreen mode

The version is advanced on the aggregate row inside the same transaction that records the Outbox event. It is not allocated from a standalone database sequence.

That distinction matters. Database sequences provide unique values, not gapless committed aggregate history. In PostgreSQL, a value obtained from nextval() is not reclaimed when the surrounding transaction rolls back. A consumer waiting for every sequence value could therefore wait forever for an event that can never exist.

The schema should protect both identities:

CREATE UNIQUE INDEX outbox_event_id_uq
    ON outbox(event_id);

CREATE UNIQUE INDEX outbox_aggregate_version_uq
    ON outbox(aggregate_id, aggregate_version);
Enter fullscreen mode Exit fullscreen mode

The first constraint protects global event identity. The second prevents two committed events from claiming the same position in one aggregate's history.

Transactional Outbox flow recording the payout update and Outbox event in one database transaction before asynchronous publication.

If the transaction fails, neither the state transition nor the event exists. If it commits, both exist with the same aggregate version. Publication becomes a recoverable asynchronous process rather than a second point of failure.

This solves the birth of the event. It does not guarantee that publication succeeds before retention expires, that a consumer applies the event, or that an external provider executes the intended effect.


6. Outbox Does Not Mean Exactly-Once

Sequence diagram showing a relay publishing an Outbox event twice after crashing before it records the first successful publication.

The duplicate here isn't evidence of a broken broker — it's the correct consequence of recovering safely from uncertainty.

Do not build correctness on the assumption that delivery happens exactly once. Build consumers so repeated delivery does not repeat the business effect.

The working invariant:

Record intent atomically. Retry delivery until it is durably observed. Give every business effect a stable identity. Reconcile every outcome that cannot be proven.

Kafka and Redpanda offer exactly-once semantics inside defined transactional boundaries. NATS offers publication deduplication within a configured window. None of that automatically makes POST https://external-bank/pay exactly-once — the broker doesn't control the bank.

Even “at least once” depends on operational assumptions: retention must not expire, durable storage must remain available, incompatible schemas must not block the consumer indefinitely, and operators must not discard unresolved records. Correctness therefore requires monitoring the age of the oldest unpublished and unprocessed intent, not merely counting successful messages.


7. Inbox Deduplication Is Atomic Only With Local Effects

Outbox solves one source of duplication, not all of them. If a client retries POST /payouts after losing the response, the service may create two payouts before the Outbox becomes relevant. Ingress commands therefore need a stable idempotency key and a stored result.

At consumption time, an Inbox prevents repeated delivery from repeating a local effect — but only when the Inbox row and that effect commit in the same database transaction.

BEGIN;

WITH accepted AS (
    INSERT INTO inbox (
        consumer_name,
        event_id,
        received_at
    )
    VALUES (
        'payout-projection',
        'evt_01K...',
        CURRENT_TIMESTAMP
    )
    ON CONFLICT (consumer_name, event_id) DO NOTHING
    RETURNING event_id
)
UPDATE payout_projection
SET status = 'APPROVED',
    source_version = 17
WHERE payout_id = 'pay_7f9q'
  AND EXISTS (SELECT 1 FROM accepted);

COMMIT;

-- Acknowledge the broker message only after COMMIT succeeds.
Enter fullscreen mode Exit fullscreen mode

The uniqueness scope includes consumer_name because different consumers may legitimately process the same event.

Transactional Inbox flow applying the deduplication record and local projection in one transaction before acknowledging the broker.

The crash behavior is now safe:

Crash point Result
Before commit Both the Inbox insert and projection update roll back; the broker redelivers
After commit Both the Inbox row and projection update exist; redelivery is a local no-op

Inbox retention must also cover the maximum broker replay and recovery horizon. Deleting deduplication rows while the corresponding events can still be replayed reopens the duplicate window.

An Inbox row is therefore a completion record only when the business effect lives inside the same local transaction.

An HTTP request, bank transfer, email, or agent tool invocation cannot join that transaction.


8. External Effects Need a Durable Operation Ledger

For an external effect, “event observed” and “business operation completed” must be separate durable facts.

If a consumer commits an Inbox row and then crashes before calling the provider, redelivery may find the event already recorded and suppress work that never happened. Inbox conflicts stay normal, the dead-letter queue remains empty, and the missing effect can look like a healthy no-op.

The consumer must atomically record recoverable work before acknowledging the event:

BEGIN;

WITH accepted AS (
    INSERT INTO inbox (
        consumer_name,
        event_id,
        received_at
    )
    VALUES (
        'payment-executor',
        'evt_01K...',
        CURRENT_TIMESTAMP
    )
    ON CONFLICT (consumer_name, event_id) DO NOTHING
    RETURNING event_id
)
INSERT INTO external_operations (
    operation_id,
    source_event_id,
    aggregate_id,
    operation_type,
    status,
    idempotency_key,
    attempt_count,
    created_at
)
SELECT
    'payout:pay_7f9q:release:decision-12',
    'evt_01K...',
    'pay_7f9q',
    'RELEASE_PAYOUT',
    'PENDING',
    'payout:pay_7f9q:release:decision-12',
    0,
    CURRENT_TIMESTAMP
FROM accepted
ON CONFLICT (operation_id) DO NOTHING;

COMMIT;

-- Acknowledge the broker event only after both durable records exist.
Enter fullscreen mode Exit fullscreen mode

The accepted dependency matters: it makes creation of the external operation conditional on this consumer accepting the event. operationId identifies the intended business effect. It should not automatically be the same as eventId: one event may legitimately produce more than one effect, while several redeliveries may refer to the same effect.

A separate worker claims durable operations using an expiring lease:

State machine for an external operation moving through PENDING, IN_FLIGHT, CONFIRMED, FAILED_TERMINAL, and UNKNOWN with reconciliation.

IN_FLIGHT is not allowed to remain permanently owned by a dead worker. When its lease expires, the operation is recovered by a scanner. If dispatch may already have started, the scanner must reconcile or retry with the same provider idempotency key — never assume that the first attempt failed.

The recovery contract must cover every crash window:

Crash point Durable state Recovery
Before the operation transaction commits No operation exists Broker redelivery recreates it
After commit, before provider dispatch PENDING or an unstarted expired claim Worker claims and dispatches it
After dispatch may have started, before local confirmation IN_FLIGHT with an ambiguous outcome Query the provider or retry with the same idempotency key
Provider response times out UNKNOWN Reconciliation resolves the outcome
After CONFIRMED Final result exists Redelivery and repeated claims are no-ops

A timeout means UNKNOWN, not FAILED. Retrying an UNKNOWN operation without provider-level uniqueness may duplicate the external effect.

A provider idempotency key is useful only within the provider's documented scope and retention window. The application's retry and reconciliation horizon must fit that contract. Some providers return the previously stored result for a repeated key; that is a provider guarantee, not a general HTTP property.

If an irreversible provider supports neither idempotent execution nor lookup by stable business identity, the caller cannot guarantee exactly-once execution. The safe choices are to require manual resolution, introduce a compensating control, or reject that provider for autonomous financial execution.

External operation recovery flow using an operation ledger, expiring worker claim, provider idempotency key, and reconciliation.


9. Ordering Requires Gap Semantics, Not Just Version Numbers

“Messages are ordered” is incomplete. The useful question is:

Ordered with respect to which aggregate, which committed history, and which consumer contract?

Processing v17 → v19 → v18 may produce an invalid state even when every message eventually arrives. An aggregateVersion makes stale events and possible gaps detectable, but the number does not enforce ordering by itself.

Comparison between committed aggregate order v17, v18, v19 and an out-of-order delivery v17, v19, v18.

A consumer can observe at least three different kinds of gap.

A temporary delivery gap

v19 arrives before committed event v18. A bounded reorder buffer may wait briefly for v18.

A permanent allocation gap

If versions come from a non-transactional database sequence, a rolled-back transaction may consume v18. The next committed event is v19, and v18 will never exist.

Aggregate versions used for contiguous ordering should therefore be advanced on the aggregate row or event stream inside the same transaction that records the event.

A legitimate consumer gap

An aggregate may produce:

v17 PayoutApproved
v18 BeneficiaryVerified
v19 PayoutReleased
Enter fullscreen mode Exit fullscreen mode

A consumer subscribed only to payout lifecycle events may observe v17 → v19. That does not prove that v18 is missing or delayed.

A consumer may require contiguous aggregate versions only when it is guaranteed to observe every committed event in that aggregate stream. A filtered consumer must instead do one of the following:

  • consume all aggregate events and no-op the irrelevant types while advancing its observed version;
  • use a consumer-specific sequence;
  • fetch the authoritative aggregate state when it observes a jump;
  • treat versions as monotonic stale-event protection without assuming contiguity.

A reorder buffer must always have explicit bounds:

MAX_GAP_AGE
MAX_BUFFERED_EVENTS_PER_AGGREGATE
MAX_BUFFERED_BYTES_PER_AGGREGATE
Enter fullscreen mode Exit fullscreen mode

When a bound is exceeded:

1. Stop applying new events for that aggregate.
2. Do not block unrelated aggregates on the same partition or shard.
3. Fetch the missing event or an authoritative snapshot.
4. Rebuild or reconcile the consumer state.
5. Escalate when the authoritative state cannot be established.
Enter fullscreen mode Exit fullscreen mode

Bounded gap recovery flow deciding whether a missing version is required and reconciling when the reorder-buffer limit expires.

A Kafka partition is a storage and ordering boundary. A NATS subject is a routing address. In either case, records for the same ordered aggregate must be routed deterministically to the same serial execution boundary.

finance.payouts.eu.s042.pk_7f9q.approved
Enter fullscreen mode Exit fullscreen mode

Parallel workers may preserve arrival order while violating completion order. If an external effect depends on order, the operation ledger must serialize or reject conflicting operations for the same aggregate.

Versions expose ordering violations. Serialization, bounded buffering, authoritative reads, and reconciliation preserve the business invariant.


10. Edge and Multiregion Systems Make Authority Explicit

A branch, regional gateway, or edge deployment that must survive a disconnected network needs a local database and local Outbox that synchronizes once connectivity returns.

Edge synchronization flow retaining durable local Outbox work while the network is unavailable and synchronizing after recovery.

A NATS leaf node does not automatically turn a local database into a conflict-free offline store. The architecture must still define: which system is authoritative per entity, how conflicts resolve, how gaps are detected, what happens when local disk fills, and how reconciliation completes.

If two regions may concurrently mutate the same aggregate, aggregateVersion can detect the conflict but cannot resolve it. The design still needs a single-writer rule, explicit ownership transfer, or a domain-specific merge protocol. “Last write wins” is not a safe default for financial state.

Local availability with eventual global convergence — not global consistency independent of the network.


11. The Reference Architecture

Reference architecture connecting legacy CDC and transactional services to a durable broker, AI decision record, deterministic policy, external operation ledger, provider, internal ledger, and reconciliation.

The essential building blocks are: an authoritative transactional source, an atomic Outbox, recoverable publication, durable transport, immutable event identity, transactional Inbox processing for local effects, bounded ordering recovery, stable decision identity, deterministic policy boundaries, a durable external-operation ledger, provider-level idempotency where available, and reconciliation where certainty cannot be obtained synchronously.


12. Event Envelopes Are Part of the Contract

{
  "eventId": "evt_01K...",
  "eventType": "PayoutApproved",
  "aggregateId": "pay_7f9q",
  "aggregateVersion": 17,
  "schemaVersion": 3,
  "occurredAt": "2026-08-07T10:57:01Z",
  "correlationId": "corr_8x7b",
  "causationId": "cmd_9c3a"
}
Enter fullscreen mode Exit fullscreen mode

eventId establishes the immutable identity of this fact. aggregateVersion identifies its committed position in the aggregate history and makes stale delivery or a possible gap detectable; it does not enforce ordering by itself.

correlationId groups a wider business flow. causationId identifies the command or event that directly caused this event. Neither should be used as an idempotency key without first defining the business operation being deduplicated.

The architecture should maintain separate identities for separate boundaries:

Identity Meaning
eventId One immutable fact
decisionRequestId One request for an assessment or decision
operationId One intended external business effect
correlationId A wider end-to-end business flow
causationId The direct cause of this record

schemaVersion also requires an explicit compatibility and migration policy. A version number without rules for backward compatibility, replay, and unsupported consumers is only metadata.


13. AI Decisions Need Stable Identity and Evidence

“Payout blocked” is not an adequate decision record. A serious system must preserve enough evidence to reconstruct why an assessment existed and how it became an authoritative action.

{
  "decisionRequestId": "dec_req_01K...",
  "sourceEventId": "evt_01K...",
  "aggregateId": "pay_7f9q",
  "aggregateVersion": 17,
  "evidenceSnapshotId": "evidence_42",
  "evidenceSnapshotHash": "sha256:...",
  "modelVersion": "risk-model-2026-08",
  "promptVersion": 4,
  "toolsetVersion": 3,
  "policyVersion": 12,
  "assessment": {
    "riskLevel": "HIGH",
    "confidence": 0.91
  },
  "createdAt": "2026-08-07T10:57:03Z"
}
Enter fullscreen mode Exit fullscreen mode

The first completed assessment for a decisionRequestId becomes the recorded result. Redelivery of the same request returns that result instead of invoking the model again.

This is decision idempotency:

Retry of the same logical request
    → return the recorded assessment

New evidence, new aggregate state, or explicit reevaluation
    → create a new decisionRequestId
Enter fullscreen mode Exit fullscreen mode

AI decision evidence chain connecting the source event, decision identity, evidence snapshot, model versions, recorded assessment, policy, and external operation identity.

The goal is not to pretend that a probabilistic model is deterministic. Exact replay may be impossible when hosted models, retrieval indexes, tools, or safety systems change. The goal is to reconstruct the inputs, versions, evidence, recorded output, policy evaluation, approvals, and executed operation.

The AI assessment is evidence, not financial authority. A deterministic policy service evaluates limits, permissions, segregation of duties, compliance rules, and approval state. Only an authorised policy result may create an external operation.

Agent tools should follow the same boundary. Read-only evidence tools may be called during assessment. Tools that create irreversible effects must be represented as authorised commands and executed through the external-operation ledger.


14. Observability Must Detect Missing Effects, Not Just Failures

A dashboard with green CPU, healthy brokers, and an empty dead-letter queue can still hide a missing payment. The most dangerous failure may look like a successful deduplication no-op.

Correctness observability needs to measure both backlog and age:

oldest_pending_outbox_age
oldest_unprocessed_event_age
oldest_pending_operation_age
oldest_in_flight_operation_age
expired_operation_claim_count
unknown_operation_age
reconciliation_backlog
reconciliation_mismatch_count
gap_buffer_oldest_age
gap_buffered_events
authorised_decisions_without_operation
confirmed_operations_without_provider_match
provider_effects_without_internal_confirmation
Enter fullscreen mode Exit fullscreen mode

Backlog size measures load. Oldest-item age measures whether the system is making progress.

Thresholds should come from business SLOs, provider idempotency windows, settlement deadlines, and the maximum acceptable uncertainty period. A universal threshold such as “five minutes” is meaningless without those contracts.

The system should periodically reconcile four sets:

Correctness reconciliation across authorised decisions, the external operation ledger, the internal financial ledger, and provider records.

Examples of correctness violations include:

  • an authorised decision with no durable external operation;
  • a PENDING operation older than its dispatch SLO;
  • an expired IN_FLIGHT claim that no worker recovered;
  • an UNKNOWN operation older than the reconciliation SLO;
  • an internally confirmed payment missing from the provider;
  • a provider-side payment with no matching internal operation;
  • an aggregate gap held beyond its configured bound.

The most dangerous state is not merely:

service = DOWN
Enter fullscreen mode Exit fullscreen mode

It is:

business_effect = UNKNOWN
Enter fullscreen mode Exit fullscreen mode

UNKNOWN must be a first-class, queryable business state with ownership, age, reconciliation policy, and escalation — not a log line inside a generic error counter.


15. Security Must Survive the Event Pipeline

Event systems copy information widely, which makes careless identifiers expensive. Don't embed account numbers, card details, or national IDs in topic or subject names — they leak into logs, metrics, traces, and dashboards.

❌ finance.account.447819002343.payment
✅ finance.payouts.eu.s042.pk_7f9q.approved
Enter fullscreen mode Exit fullscreen mode

An authenticated event is not permanent authorization to repeat an action. Replays and delayed deliveries must pass the current execution contract or an explicitly versioned historical authorization contract.

Agents should not hold unrestricted provider credentials. The execution worker should receive narrowly scoped authority to perform one approved operationId, with amount, currency, beneficiary, policy decision, and validity window bound to that operation.

Encryption in transit and at rest, fine-grained access control, schema governance, controlled replay, dead-letter handling, and data residency controls are non-negotiable. Durability without governance only makes mistakes survive longer.


16. Choose the Mechanism From the Invariant

The architecture should follow the invariant. The invariant should not follow the product logo.

Required invariant Mechanism
Business state and publication intent are born together Transactional Outbox
A local projection is not applied twice Inbox and local effect in one transaction
An external effect survives executor crashes Durable external-operation ledger
Retried provider calls do not repeat an effect Stable operationId and provider idempotency contract
Ambiguous provider outcomes become resolvable UNKNOWN state and reconciliation
Aggregate transitions do not complete out of order Deterministic routing and serial execution per aggregate
A missing version cannot block forever Bounded buffer and authoritative reconciliation
A transport retry does not create another AI decision Stable decisionRequestId and stored assessment
Probabilistic reasoning cannot directly move money Deterministic policy and approval boundary
Regional autonomy does not create silent conflict Explicit authority, ownership transfer, or merge protocol
Replay does not bypass governance Versioned authorization, schema, retention, and access policy

Products implement parts of these mechanisms. They do not define the business invariant.


17. Exactly-Once Requires an Explicit Boundary

“Does this broker support exactly-once?” is rarely the useful question.

The useful questions are:

  • Exactly once inside which transaction boundary?
  • Which system assigns the business effect its identity?
  • Which component rejects a repeated identity?
  • How long is that identity retained?
  • Can an ambiguous result be queried?
  • What happens when certainty cannot be recovered automatically?

Kafka can provide exactly-once processing across supported Kafka operations because the relevant offsets, state, and produced records participate in the Kafka transaction boundary. That guarantee does not automatically include an unrelated database or external provider.

For irreversible effects, correctness contains both safety and liveness:

Safety:
The same operationId must not create two business effects.

Liveness:
Every authorised operation must eventually become
CONFIRMED, FAILED_TERMINAL, or explicitly escalated.
Enter fullscreen mode Exit fullscreen mode

Deduplication without liveness is not sufficient. An Inbox tombstone that suppresses work which never reached the provider may prevent a duplicate while permanently losing the intended effect.

The end-to-end contract is therefore:

Record each business intent atomically.
Deliver until the intent is durably observed.
Create one stable identity per intended effect.
Use destination-level idempotency where available.
Never interpret timeout as definitive failure.
Reconcile every ambiguous outcome.
Escalate when the architecture cannot prove the result.
Enter fullscreen mode Exit fullscreen mode

Exactly-once is not merely a transport property. It is a claim that must name its boundary, identity, enforcement point, retention window, and recovery protocol.

If an irreversible provider offers neither idempotent execution nor lookup by stable business identity, the architecture must not claim exactly-once execution across that boundary.


18. What AI Changes

AI does not repeal distributed systems. Agents still run on networks. Tools still time out. Databases still commit independently. Messages can still be duplicated.

AI also introduces a probabilistic decision boundary between fact and effect. That boundary needs its own durable identity. Reprocessing an event must not silently create another authoritative assessment, while a legitimate reevaluation based on new evidence must remain possible and auditable.

Comparison between a traditional event-to-projection path and an agentic path containing reasoning, tool selection, deterministic policy, and external execution.

What AI changes is the distance between information and action and the number of probabilistic steps inside that distance.

As agents gain operational authority, distributed-systems correctness becomes more important, not less. Agent reliability is not primarily a prompt-engineering problem. The decisive guarantees still live in atomicity, stable identity, bounded recovery, deterministic authorization, least privilege, reconciliation, and auditability.

The model is one component. It is not the transaction boundary, the source of financial truth, or the enforcement point for an irreversible effect.


Conclusion

Transactional Outbox makes business state and publication intent atomic at the source. A relay or CDC process transports that intent, while durable delivery and transactional Inbox processing make local consumers tolerant of redelivery.

Those guarantees stop at the local database boundary. External effects require their own durable operation identity, recoverable claim state, provider-level uniqueness contract, and reconciliation. Aggregate versions expose stale events and possible gaps; deterministic routing, bounded buffering, and authoritative recovery protect the ordering invariant. AI assessments require stable decision identity, evidence snapshots, and deterministic policy approval before they can create an external operation.

AI agents do not create these distributed systems problems. They make ignoring them more expensive.

A duplicate event can now travel farther. A retry can activate more capable software. An uncertain result can trigger decisions that affect customers, accounts, infrastructure, or money.

That is why modern agentic architecture should not begin by asking which model is smartest or which broker is fastest. It should begin with the business invariant:

Every financial movement must have a stable operation identity, originate from an authorised and versioned business state, survive publication and executor failures, tolerate redelivery, detect ordering violations, pass deterministic policy controls, use destination-level uniqueness where available, and remain reconcilable when certainty cannot be obtained synchronously.

Or, simply:

Record intent once. Give every effect an identity. Reconcile uncertainty.

Then ask the question that matters when everything appears healthy and the acknowledgement never arrives:

Can your system prove that the same payment will not leave twice?

And can it also prove that an authorised payment will not disappear behind a successful deduplication record?

A correct design needs both guarantees: no duplicate effect and no silently lost intent.

Not whether Kafka stayed online. Not whether the database stayed online. Not whether your AI agent produced the right explanation.

Whether the architecture preserved both safety and liveness when the network stopped being able to tell you what happened — and whether every unresolved outcome remained visible until reality could be established.


Revision note — August 2026: Expanded to cover recoverable external-effect claims and bounded reorder buffers after thoughtful reader feedback.

Technical References

Top comments (1)

Collapse
 
anp2network profile image
ANP2 Network

One sharp edge in the inbox pattern is that the ON CONFLICT DO NOTHING transaction only closes the loop when the effect lives in the same database as the inbox row. Section 8's external executor cannot join that transaction, so if the inbox row commits and the process dies before the provider call is attempted, the event is permanently marked seen and the effect is silently lost. That is worse than the duplicate the inbox exists to prevent. Inbox conflicts stay flat and the dead-letter queue stays empty, so the miss reads as a healthy no-op and none of the correctness metrics move. For effects that leave the process the dedup row wants to be a claim row, CLAIMED -> SENT -> CONFIRMED/UNKNOWN, written before dispatch and advanced by the same UNKNOWN reconciliation loop you already have, so a crash leaves recoverable work instead of a tombstone that suppresses the retry.

The reorder buffer needs a bound for a related reason. Gaps in aggregateVersion can be permanent when versions come off a database sequence, since a rolled back transaction burns v18 and v19 then waits behind a number that will never arrive, which turns an ordering guard into an availability outage on one aggregate. Your outbox SQL avoids that by taking version 17 off the payout row instead of a sequence, worth saying out loud because most implementations reach for the sequence first. Consumers that filter event types still see legitimate gaps, so the buffer needs a timeout that escalates into reconciliation instead of holding.