The dual-write problem existed long before LLMs. Now a duplicate event can trigger fraud investigations, payment holds, refunds, or financial execution.
At 11:57 PM on Black Friday, a global commerce platform approves a €2.4 million payout to a merchant.
The ledger commits the transaction. The application publishes PayoutApproved. Then the request times out.
Nothing is necessarily down. The database is healthy. The broker may also be healthy. The network recovered milliseconds later. Every dashboard is still green.
Yet the system has lost something more dangerous than availability.
It has lost certainty.
Did the broker receive the event?
If the producer does nothing, the risk system may never analyze the payout. If the producer retries — and the first publication actually succeeded — the event may be delivered twice. If downstream execution is not idempotent, the same business operation may be applied twice.
A few milliseconds of uncertainty can become a multimillion-euro question:
Did the operation fail, or did we merely fail to observe that it succeeded?
This is not fundamentally a Kafka problem, a NATS problem, or an AI problem. It is a distributed systems problem.
Modern AI agents make it more expensive because event consumers are becoming more capable. A duplicate event no longer has to mean a duplicated projection or a repeated log line. It may wake an agent that assesses fraud, recommends a payment hold, launches an investigation, requests a refund, or calls a tool connected to an external financial system.
The delivery semantics did not get worse. What sits behind the event became more powerful.
The architecture needs answers to six questions:
- Where does the event become authoritative?
- How is publication recovered after failure?
- How do consumers tolerate redelivery?
- How is ordering preserved where the business requires it?
- How does each irreversible effect receive a stable identity, survive executor crashes, and remain reconcilable when its outcome is unknown?
- How does a probabilistic assessment become one authoritative, deterministic decision?
1. Your Broker Does Not Solve the Dual-Write Problem
A service that approves a payout typically performs two operations that conceptually belong together but live in two separate transactional systems: the database and the broker.
There is no automatic atomic transaction spanning both systems, which creates three distinct failure states:
| Failure | Consequence |
|---|---|
| Database commits, event publish fails | Business state exists, downstream systems never learn about it |
| Event published, database transaction rolls back | Consumers observe something that never became true |
| Broker accepts event, ACK is lost, producer retries | Possible duplicate delivery |
This is the dual-write problem. A broker gives you decoupling, buffering, durability, fan-out, replay, and failure isolation. What it does not give you is atomicity between an event and the database state that caused that event to exist. That distinction is the foundation for everything that follows.
2. Broker Guarantees Stop at Their Transaction Boundary
The useful question is not “which broker is best?” It is:
Which state transitions can this product make atomic, and where does that atomicity stop?
Kafka and Redpanda can provide transactional guarantees across supported broker operations. Kafka Streams, for example, can atomically combine consumed offsets, state-store updates, and produced records within the Kafka transaction boundary.
RabbitMQ provides durable queues, publisher confirms, acknowledgements, quorum queues, and routing mechanisms. NATS JetStream provides durable streams, acknowledgements, redelivery, and time-bounded publication deduplication.
These are valuable guarantees, but they belong to different boundaries:
| Boundary | Guarantee that may be available |
|---|---|
| Broker record → broker record | Broker transaction or atomic stream-processing operation |
| Producer → broker | Confirmed or acknowledged publication |
| Broker → consumer | Durable delivery with acknowledgement and redelivery |
| Application database → broker | Not atomic without an Outbox or equivalent mechanism |
| Broker → external provider | Not atomic |
| Internal ledger → bank transfer | Not atomic without a provider-level uniqueness and reconciliation contract |
A broker transaction cannot atomically commit an unrelated Postgres transaction, a payment-provider request, or a bank transfer.
Choose the broker from the workload. Design correctness from the transaction boundary.
3. AI Agents Don't Create the Problem — They Increase Its Blast Radius
The dual-write problem predates LLMs by decades. Retries, duplicate messages, lost acknowledgements, and reordered events are established distributed systems failure modes. What's changed is what a consumer can now do with a duplicate.
If the old projection consumer processes a duplicate twice, and the projection is idempotent, the impact may be negligible. If the same duplicate now traverses an agentic path, it can trigger a second fraud investigation, a second account block, or a second refund instruction.
That doesn't make AI agents inherently unsafe. It means the architecture around them must assume messages arrive more than once, agents can be wrong, tools can time out, models can change, and external systems can return ambiguous outcomes.
Once an agent participates, the architecture must distinguish three identities:
-
eventId— the immutable identity of a fact that happened; -
decisionRequestId— the identity of one request for an assessment or decision; -
operationId— the identity of one intended external business effect.
These identifiers are related through causation, but they are not interchangeable. One event may legitimately trigger multiple decisions, and one approved decision may produce multiple external operations. Reusing eventId for every boundary can suppress legitimate work or accidentally collapse distinct effects into one.
An AI agent should never move money simply because it generated a plausible answer. The safer boundary separates reasoning from execution:
AI recommends. Policies decide. Systems execute.
The agent analyzes, classifies, explains, and recommends. A deterministic service evaluates limits, permissions, segregation of duties, and compliance. Only then does an idempotent executor perform the external effect.
A retry of the same decisionRequestId should return the already-recorded assessment. A reevaluation caused by new evidence, a new aggregate version, or an explicit review request receives a new identity. Idempotency here does not require a probabilistic model to generate identical tokens; it prevents a transport retry from creating a second authoritative decision.
4. CDC Is More Than a Legacy Escape Hatch
Enterprise systems rarely start clean. Banks, insurers, and telecoms often depend on ERPs and settlement engines built years before event-driven architecture became mainstream. Rewriting them to emit domain events may be too risky.
Change Data Capture — commonly Debezium — observes transactional database logs and captures committed changes without touching the legacy application.
The trap: a row mutation is not automatically a domain event. STATUS = 'A' might mean approved, awaiting review, or an internal transition with no meaning outside the legacy system. If downstream services must understand physical tables and cryptic status codes, the database schema has silently become a public contract — and a fragile one.
A translation layer protects the rest of the organization from the legacy system's physical representation. CDC is also strategic beyond migration: it can feed search indexes, warehouses, read models — and most importantly, it can capture an Outbox table, transporting events the application deliberately designed rather than technical noise.
CDC also inherits the ordering and transaction semantics of the source log. A translation layer must preserve transaction identity and commit order where the downstream business invariant depends on them; row-level change order alone is not automatically domain-event order.
5. Transactional Outbox Fixes the Birth of the Event
A new payment service knows two facts at the same instant: the payout changed state, and an event describing that change must eventually be published. Instead of two independent writes, it records both facts in one local transaction.
BEGIN;
WITH advanced_payout AS (
UPDATE payouts
SET status = 'APPROVED',
version = version + 1
WHERE id = 'pay_7f9q'
AND version = 16
RETURNING id, version
)
INSERT INTO outbox (
event_id,
aggregate_id,
aggregate_version,
event_type,
payload,
occurred_at
)
SELECT
'evt_01K...',
id,
version,
'PayoutApproved',
'{"payoutId":"pay_7f9q"}',
CURRENT_TIMESTAMP
FROM advanced_payout
RETURNING aggregate_version;
-- The application must require exactly one returned row.
-- Zero rows means that the expected version was stale.
COMMIT;
The version is advanced on the aggregate row inside the same transaction that records the Outbox event. It is not allocated from a standalone database sequence.
That distinction matters. Database sequences provide unique values, not gapless committed aggregate history. In PostgreSQL, a value obtained from nextval() is not reclaimed when the surrounding transaction rolls back. A consumer waiting for every sequence value could therefore wait forever for an event that can never exist.
The schema should protect both identities:
CREATE UNIQUE INDEX outbox_event_id_uq
ON outbox(event_id);
CREATE UNIQUE INDEX outbox_aggregate_version_uq
ON outbox(aggregate_id, aggregate_version);
The first constraint protects global event identity. The second prevents two committed events from claiming the same position in one aggregate's history.
If the transaction fails, neither the state transition nor the event exists. If it commits, both exist with the same aggregate version. Publication becomes a recoverable asynchronous process rather than a second point of failure.
This solves the birth of the event. It does not guarantee that publication succeeds before retention expires, that a consumer applies the event, or that an external provider executes the intended effect.
6. Outbox Does Not Mean Exactly-Once
The duplicate here isn't evidence of a broken broker — it's the correct consequence of recovering safely from uncertainty.
Do not build correctness on the assumption that delivery happens exactly once. Build consumers so repeated delivery does not repeat the business effect.
The working invariant:
Record intent atomically. Retry delivery until it is durably observed. Give every business effect a stable identity. Reconcile every outcome that cannot be proven.
Kafka and Redpanda offer exactly-once semantics inside defined transactional boundaries. NATS offers publication deduplication within a configured window. None of that automatically makes POST https://external-bank/pay exactly-once — the broker doesn't control the bank.
Even “at least once” depends on operational assumptions: retention must not expire, durable storage must remain available, incompatible schemas must not block the consumer indefinitely, and operators must not discard unresolved records. Correctness therefore requires monitoring the age of the oldest unpublished and unprocessed intent, not merely counting successful messages.
7. Inbox Deduplication Is Atomic Only With Local Effects
Outbox solves one source of duplication, not all of them. If a client retries POST /payouts after losing the response, the service may create two payouts before the Outbox becomes relevant. Ingress commands therefore need a stable idempotency key and a stored result.
At consumption time, an Inbox prevents repeated delivery from repeating a local effect — but only when the Inbox row and that effect commit in the same database transaction.
BEGIN;
WITH accepted AS (
INSERT INTO inbox (
consumer_name,
event_id,
received_at
)
VALUES (
'payout-projection',
'evt_01K...',
CURRENT_TIMESTAMP
)
ON CONFLICT (consumer_name, event_id) DO NOTHING
RETURNING event_id
)
UPDATE payout_projection
SET status = 'APPROVED',
source_version = 17
WHERE payout_id = 'pay_7f9q'
AND EXISTS (SELECT 1 FROM accepted);
COMMIT;
-- Acknowledge the broker message only after COMMIT succeeds.
The uniqueness scope includes consumer_name because different consumers may legitimately process the same event.
The crash behavior is now safe:
| Crash point | Result |
|---|---|
| Before commit | Both the Inbox insert and projection update roll back; the broker redelivers |
| After commit | Both the Inbox row and projection update exist; redelivery is a local no-op |
Inbox retention must also cover the maximum broker replay and recovery horizon. Deleting deduplication rows while the corresponding events can still be replayed reopens the duplicate window.
An Inbox row is therefore a completion record only when the business effect lives inside the same local transaction.
An HTTP request, bank transfer, email, or agent tool invocation cannot join that transaction.
8. External Effects Need a Durable Operation Ledger
For an external effect, “event observed” and “business operation completed” must be separate durable facts.
If a consumer commits an Inbox row and then crashes before calling the provider, redelivery may find the event already recorded and suppress work that never happened. Inbox conflicts stay normal, the dead-letter queue remains empty, and the missing effect can look like a healthy no-op.
The consumer must atomically record recoverable work before acknowledging the event:
BEGIN;
WITH accepted AS (
INSERT INTO inbox (
consumer_name,
event_id,
received_at
)
VALUES (
'payment-executor',
'evt_01K...',
CURRENT_TIMESTAMP
)
ON CONFLICT (consumer_name, event_id) DO NOTHING
RETURNING event_id
)
INSERT INTO external_operations (
operation_id,
source_event_id,
aggregate_id,
operation_type,
status,
idempotency_key,
attempt_count,
created_at
)
SELECT
'payout:pay_7f9q:release:decision-12',
'evt_01K...',
'pay_7f9q',
'RELEASE_PAYOUT',
'PENDING',
'payout:pay_7f9q:release:decision-12',
0,
CURRENT_TIMESTAMP
FROM accepted
ON CONFLICT (operation_id) DO NOTHING;
COMMIT;
-- Acknowledge the broker event only after both durable records exist.
The accepted dependency matters: it makes creation of the external operation conditional on this consumer accepting the event. operationId identifies the intended business effect. It should not automatically be the same as eventId: one event may legitimately produce more than one effect, while several redeliveries may refer to the same effect.
A separate worker claims durable operations using an expiring lease:
IN_FLIGHT is not allowed to remain permanently owned by a dead worker. When its lease expires, the operation is recovered by a scanner. If dispatch may already have started, the scanner must reconcile or retry with the same provider idempotency key — never assume that the first attempt failed.
The recovery contract must cover every crash window:
| Crash point | Durable state | Recovery |
|---|---|---|
| Before the operation transaction commits | No operation exists | Broker redelivery recreates it |
| After commit, before provider dispatch |
PENDING or an unstarted expired claim |
Worker claims and dispatches it |
| After dispatch may have started, before local confirmation |
IN_FLIGHT with an ambiguous outcome |
Query the provider or retry with the same idempotency key |
| Provider response times out | UNKNOWN |
Reconciliation resolves the outcome |
After CONFIRMED
|
Final result exists | Redelivery and repeated claims are no-ops |
A timeout means UNKNOWN, not FAILED. Retrying an UNKNOWN operation without provider-level uniqueness may duplicate the external effect.
A provider idempotency key is useful only within the provider's documented scope and retention window. The application's retry and reconciliation horizon must fit that contract. Some providers return the previously stored result for a repeated key; that is a provider guarantee, not a general HTTP property.
If an irreversible provider supports neither idempotent execution nor lookup by stable business identity, the caller cannot guarantee exactly-once execution. The safe choices are to require manual resolution, introduce a compensating control, or reject that provider for autonomous financial execution.
9. Ordering Requires Gap Semantics, Not Just Version Numbers
“Messages are ordered” is incomplete. The useful question is:
Ordered with respect to which aggregate, which committed history, and which consumer contract?
Processing v17 → v19 → v18 may produce an invalid state even when every message eventually arrives. An aggregateVersion makes stale events and possible gaps detectable, but the number does not enforce ordering by itself.
A consumer can observe at least three different kinds of gap.
A temporary delivery gap
v19 arrives before committed event v18. A bounded reorder buffer may wait briefly for v18.
A permanent allocation gap
If versions come from a non-transactional database sequence, a rolled-back transaction may consume v18. The next committed event is v19, and v18 will never exist.
Aggregate versions used for contiguous ordering should therefore be advanced on the aggregate row or event stream inside the same transaction that records the event.
A legitimate consumer gap
An aggregate may produce:
v17 PayoutApproved
v18 BeneficiaryVerified
v19 PayoutReleased
A consumer subscribed only to payout lifecycle events may observe v17 → v19. That does not prove that v18 is missing or delayed.
A consumer may require contiguous aggregate versions only when it is guaranteed to observe every committed event in that aggregate stream. A filtered consumer must instead do one of the following:
- consume all aggregate events and no-op the irrelevant types while advancing its observed version;
- use a consumer-specific sequence;
- fetch the authoritative aggregate state when it observes a jump;
- treat versions as monotonic stale-event protection without assuming contiguity.
A reorder buffer must always have explicit bounds:
MAX_GAP_AGE
MAX_BUFFERED_EVENTS_PER_AGGREGATE
MAX_BUFFERED_BYTES_PER_AGGREGATE
When a bound is exceeded:
1. Stop applying new events for that aggregate.
2. Do not block unrelated aggregates on the same partition or shard.
3. Fetch the missing event or an authoritative snapshot.
4. Rebuild or reconcile the consumer state.
5. Escalate when the authoritative state cannot be established.
A Kafka partition is a storage and ordering boundary. A NATS subject is a routing address. In either case, records for the same ordered aggregate must be routed deterministically to the same serial execution boundary.
finance.payouts.eu.s042.pk_7f9q.approved
Parallel workers may preserve arrival order while violating completion order. If an external effect depends on order, the operation ledger must serialize or reject conflicting operations for the same aggregate.
Versions expose ordering violations. Serialization, bounded buffering, authoritative reads, and reconciliation preserve the business invariant.
10. Edge and Multiregion Systems Make Authority Explicit
A branch, regional gateway, or edge deployment that must survive a disconnected network needs a local database and local Outbox that synchronizes once connectivity returns.
A NATS leaf node does not automatically turn a local database into a conflict-free offline store. The architecture must still define: which system is authoritative per entity, how conflicts resolve, how gaps are detected, what happens when local disk fills, and how reconciliation completes.
If two regions may concurrently mutate the same aggregate, aggregateVersion can detect the conflict but cannot resolve it. The design still needs a single-writer rule, explicit ownership transfer, or a domain-specific merge protocol. “Last write wins” is not a safe default for financial state.
Local availability with eventual global convergence — not global consistency independent of the network.
11. The Reference Architecture
The essential building blocks are: an authoritative transactional source, an atomic Outbox, recoverable publication, durable transport, immutable event identity, transactional Inbox processing for local effects, bounded ordering recovery, stable decision identity, deterministic policy boundaries, a durable external-operation ledger, provider-level idempotency where available, and reconciliation where certainty cannot be obtained synchronously.
12. Event Envelopes Are Part of the Contract
{
"eventId": "evt_01K...",
"eventType": "PayoutApproved",
"aggregateId": "pay_7f9q",
"aggregateVersion": 17,
"schemaVersion": 3,
"occurredAt": "2026-08-07T10:57:01Z",
"correlationId": "corr_8x7b",
"causationId": "cmd_9c3a"
}
eventId establishes the immutable identity of this fact. aggregateVersion identifies its committed position in the aggregate history and makes stale delivery or a possible gap detectable; it does not enforce ordering by itself.
correlationId groups a wider business flow. causationId identifies the command or event that directly caused this event. Neither should be used as an idempotency key without first defining the business operation being deduplicated.
The architecture should maintain separate identities for separate boundaries:
| Identity | Meaning |
|---|---|
eventId |
One immutable fact |
decisionRequestId |
One request for an assessment or decision |
operationId |
One intended external business effect |
correlationId |
A wider end-to-end business flow |
causationId |
The direct cause of this record |
schemaVersion also requires an explicit compatibility and migration policy. A version number without rules for backward compatibility, replay, and unsupported consumers is only metadata.
13. AI Decisions Need Stable Identity and Evidence
“Payout blocked” is not an adequate decision record. A serious system must preserve enough evidence to reconstruct why an assessment existed and how it became an authoritative action.
{
"decisionRequestId": "dec_req_01K...",
"sourceEventId": "evt_01K...",
"aggregateId": "pay_7f9q",
"aggregateVersion": 17,
"evidenceSnapshotId": "evidence_42",
"evidenceSnapshotHash": "sha256:...",
"modelVersion": "risk-model-2026-08",
"promptVersion": 4,
"toolsetVersion": 3,
"policyVersion": 12,
"assessment": {
"riskLevel": "HIGH",
"confidence": 0.91
},
"createdAt": "2026-08-07T10:57:03Z"
}
The first completed assessment for a decisionRequestId becomes the recorded result. Redelivery of the same request returns that result instead of invoking the model again.
This is decision idempotency:
Retry of the same logical request
→ return the recorded assessment
New evidence, new aggregate state, or explicit reevaluation
→ create a new decisionRequestId
The goal is not to pretend that a probabilistic model is deterministic. Exact replay may be impossible when hosted models, retrieval indexes, tools, or safety systems change. The goal is to reconstruct the inputs, versions, evidence, recorded output, policy evaluation, approvals, and executed operation.
The AI assessment is evidence, not financial authority. A deterministic policy service evaluates limits, permissions, segregation of duties, compliance rules, and approval state. Only an authorised policy result may create an external operation.
Agent tools should follow the same boundary. Read-only evidence tools may be called during assessment. Tools that create irreversible effects must be represented as authorised commands and executed through the external-operation ledger.
14. Observability Must Detect Missing Effects, Not Just Failures
A dashboard with green CPU, healthy brokers, and an empty dead-letter queue can still hide a missing payment. The most dangerous failure may look like a successful deduplication no-op.
Correctness observability needs to measure both backlog and age:
oldest_pending_outbox_age
oldest_unprocessed_event_age
oldest_pending_operation_age
oldest_in_flight_operation_age
expired_operation_claim_count
unknown_operation_age
reconciliation_backlog
reconciliation_mismatch_count
gap_buffer_oldest_age
gap_buffered_events
authorised_decisions_without_operation
confirmed_operations_without_provider_match
provider_effects_without_internal_confirmation
Backlog size measures load. Oldest-item age measures whether the system is making progress.
Thresholds should come from business SLOs, provider idempotency windows, settlement deadlines, and the maximum acceptable uncertainty period. A universal threshold such as “five minutes” is meaningless without those contracts.
The system should periodically reconcile four sets:
Examples of correctness violations include:
- an authorised decision with no durable external operation;
- a
PENDINGoperation older than its dispatch SLO; - an expired
IN_FLIGHTclaim that no worker recovered; - an
UNKNOWNoperation older than the reconciliation SLO; - an internally confirmed payment missing from the provider;
- a provider-side payment with no matching internal operation;
- an aggregate gap held beyond its configured bound.
The most dangerous state is not merely:
service = DOWN
It is:
business_effect = UNKNOWN
UNKNOWN must be a first-class, queryable business state with ownership, age, reconciliation policy, and escalation — not a log line inside a generic error counter.
15. Security Must Survive the Event Pipeline
Event systems copy information widely, which makes careless identifiers expensive. Don't embed account numbers, card details, or national IDs in topic or subject names — they leak into logs, metrics, traces, and dashboards.
❌ finance.account.447819002343.payment
✅ finance.payouts.eu.s042.pk_7f9q.approved
An authenticated event is not permanent authorization to repeat an action. Replays and delayed deliveries must pass the current execution contract or an explicitly versioned historical authorization contract.
Agents should not hold unrestricted provider credentials. The execution worker should receive narrowly scoped authority to perform one approved operationId, with amount, currency, beneficiary, policy decision, and validity window bound to that operation.
Encryption in transit and at rest, fine-grained access control, schema governance, controlled replay, dead-letter handling, and data residency controls are non-negotiable. Durability without governance only makes mistakes survive longer.
16. Choose the Mechanism From the Invariant
The architecture should follow the invariant. The invariant should not follow the product logo.
| Required invariant | Mechanism |
|---|---|
| Business state and publication intent are born together | Transactional Outbox |
| A local projection is not applied twice | Inbox and local effect in one transaction |
| An external effect survives executor crashes | Durable external-operation ledger |
| Retried provider calls do not repeat an effect | Stable operationId and provider idempotency contract |
| Ambiguous provider outcomes become resolvable |
UNKNOWN state and reconciliation |
| Aggregate transitions do not complete out of order | Deterministic routing and serial execution per aggregate |
| A missing version cannot block forever | Bounded buffer and authoritative reconciliation |
| A transport retry does not create another AI decision | Stable decisionRequestId and stored assessment |
| Probabilistic reasoning cannot directly move money | Deterministic policy and approval boundary |
| Regional autonomy does not create silent conflict | Explicit authority, ownership transfer, or merge protocol |
| Replay does not bypass governance | Versioned authorization, schema, retention, and access policy |
Products implement parts of these mechanisms. They do not define the business invariant.
17. Exactly-Once Requires an Explicit Boundary
“Does this broker support exactly-once?” is rarely the useful question.
The useful questions are:
- Exactly once inside which transaction boundary?
- Which system assigns the business effect its identity?
- Which component rejects a repeated identity?
- How long is that identity retained?
- Can an ambiguous result be queried?
- What happens when certainty cannot be recovered automatically?
Kafka can provide exactly-once processing across supported Kafka operations because the relevant offsets, state, and produced records participate in the Kafka transaction boundary. That guarantee does not automatically include an unrelated database or external provider.
For irreversible effects, correctness contains both safety and liveness:
Safety:
The same operationId must not create two business effects.
Liveness:
Every authorised operation must eventually become
CONFIRMED, FAILED_TERMINAL, or explicitly escalated.
Deduplication without liveness is not sufficient. An Inbox tombstone that suppresses work which never reached the provider may prevent a duplicate while permanently losing the intended effect.
The end-to-end contract is therefore:
Record each business intent atomically.
Deliver until the intent is durably observed.
Create one stable identity per intended effect.
Use destination-level idempotency where available.
Never interpret timeout as definitive failure.
Reconcile every ambiguous outcome.
Escalate when the architecture cannot prove the result.
Exactly-once is not merely a transport property. It is a claim that must name its boundary, identity, enforcement point, retention window, and recovery protocol.
If an irreversible provider offers neither idempotent execution nor lookup by stable business identity, the architecture must not claim exactly-once execution across that boundary.
18. What AI Changes
AI does not repeal distributed systems. Agents still run on networks. Tools still time out. Databases still commit independently. Messages can still be duplicated.
AI also introduces a probabilistic decision boundary between fact and effect. That boundary needs its own durable identity. Reprocessing an event must not silently create another authoritative assessment, while a legitimate reevaluation based on new evidence must remain possible and auditable.
What AI changes is the distance between information and action and the number of probabilistic steps inside that distance.
As agents gain operational authority, distributed-systems correctness becomes more important, not less. Agent reliability is not primarily a prompt-engineering problem. The decisive guarantees still live in atomicity, stable identity, bounded recovery, deterministic authorization, least privilege, reconciliation, and auditability.
The model is one component. It is not the transaction boundary, the source of financial truth, or the enforcement point for an irreversible effect.
Conclusion
Transactional Outbox makes business state and publication intent atomic at the source. A relay or CDC process transports that intent, while durable delivery and transactional Inbox processing make local consumers tolerant of redelivery.
Those guarantees stop at the local database boundary. External effects require their own durable operation identity, recoverable claim state, provider-level uniqueness contract, and reconciliation. Aggregate versions expose stale events and possible gaps; deterministic routing, bounded buffering, and authoritative recovery protect the ordering invariant. AI assessments require stable decision identity, evidence snapshots, and deterministic policy approval before they can create an external operation.
AI agents do not create these distributed systems problems. They make ignoring them more expensive.
A duplicate event can now travel farther. A retry can activate more capable software. An uncertain result can trigger decisions that affect customers, accounts, infrastructure, or money.
That is why modern agentic architecture should not begin by asking which model is smartest or which broker is fastest. It should begin with the business invariant:
Every financial movement must have a stable operation identity, originate from an authorised and versioned business state, survive publication and executor failures, tolerate redelivery, detect ordering violations, pass deterministic policy controls, use destination-level uniqueness where available, and remain reconcilable when certainty cannot be obtained synchronously.
Or, simply:
Record intent once. Give every effect an identity. Reconcile uncertainty.
Then ask the question that matters when everything appears healthy and the acknowledgement never arrives:
Can your system prove that the same payment will not leave twice?
And can it also prove that an authorised payment will not disappear behind a successful deduplication record?
A correct design needs both guarantees: no duplicate effect and no silently lost intent.
Not whether Kafka stayed online. Not whether the database stayed online. Not whether your AI agent produced the right explanation.
Whether the architecture preserved both safety and liveness when the network stopped being able to tell you what happened — and whether every unresolved outcome remained visible until reality could be established.
Revision note — August 2026: Expanded to cover recoverable external-effect claims and bounded reorder buffers after thoughtful reader feedback.
Technical References
- Apache Kafka Documentation
- Apache Kafka 4.0 — ZooKeeper Removal, KRaft-only Architecture
- Apache Kafka Streams — Exactly-Once Processing Boundary
- Debezium Outbox Event Router
- Debezium Server + NATS JetStream Support
- NATS JetStream Delivery and Acknowledgements
- NATS JetStream Publication Deduplication
- NATS Deterministic Subject Partitioning
- NATS Leaf Nodes and JetStream Domains
- PostgreSQL Sequence Functions — Gaps and Rollbacks
- RabbitMQ Queues, Quorum Queues, and Streams
- Redpanda Transactions and Kafka Compatibility
- Stripe API — Idempotent Requests

















Top comments (1)
One sharp edge in the inbox pattern is that the ON CONFLICT DO NOTHING transaction only closes the loop when the effect lives in the same database as the inbox row. Section 8's external executor cannot join that transaction, so if the inbox row commits and the process dies before the provider call is attempted, the event is permanently marked seen and the effect is silently lost. That is worse than the duplicate the inbox exists to prevent. Inbox conflicts stay flat and the dead-letter queue stays empty, so the miss reads as a healthy no-op and none of the correctness metrics move. For effects that leave the process the dedup row wants to be a claim row, CLAIMED -> SENT -> CONFIRMED/UNKNOWN, written before dispatch and advanced by the same UNKNOWN reconciliation loop you already have, so a crash leaves recoverable work instead of a tombstone that suppresses the retry.
The reorder buffer needs a bound for a related reason. Gaps in aggregateVersion can be permanent when versions come off a database sequence, since a rolled back transaction burns v18 and v19 then waits behind a number that will never arrive, which turns an ordering guard into an availability outage on one aggregate. Your outbox SQL avoids that by taking version 17 off the payout row instead of a sequence, worth saying out loud because most implementations reach for the sequence first. Consumers that filter event types still see legitimate gaps, so the buffer needs a timeout that escalates into reconciliation instead of holding.