The Workflow That Sent It Twice: Reconciling Side Effects After a Retry in n8n

This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is a failure I actually watched happen, not a hypothetical.

The workflow was simple, and that was the problem. A new row appeared in a sheet, n8n picked it up, and it asked a provider to send one notification to one customer. One row, one send, no ambiguity — until the day the provider was slow.

A failure you can reproduce

Here is the exact shape of it. The workflow reads a row, builds a message, and calls the provider's send endpoint. The provider accepts the message, queues it, and then the connection stalls before the response comes back. n8n sees a timeout, not a success and not a clean failure. Its retry policy does what it was told to do: it runs the step again.

The second call is a brand-new request as far as the provider is concerned. Same customer, same message, no shared identifier. The provider accepts that one too, and this time the response comes back promptly. The workflow marks the run green. The customer receives the message twice.

Nothing in the logs looks wrong on the second attempt, because nothing was wrong with the second attempt in isolation. The bug is not in the retry. The bug is that the workflow had no memory of the first attempt, and no way to ask the provider what had already happened.

You can reproduce this without any production traffic. Point a test workflow at a mock endpoint that accepts the first request, delays past your timeout, and then returns success on the second request. If your workflow sends twice, you have found the bug before a customer does. If it sends once, check whether that is because of a real reconciliation step, or merely because your mock returned an error before accepting anything. Those are very different outcomes.

Accepted is not the same as confirmed

Most retry bugs come from collapsing three states into two. A workflow usually knows only success and failure. A side effect — anything the outside world can observe, such as a message, a charge, a shipment label, or a calendar invite — needs three states:

• Not sent. No request reached the provider, or the provider rejected it before recording anything. Safe to send.
• Accepted. The provider recorded the effect, but the workflow never saw the confirmation. The response was lost, timed out, or the workflow crashed after the provider committed. Not safe to send again without checking.
• Confirmed. The workflow holds the provider's record identifier and has verified it. Safe to skip on retry.

A timeout lives in the gap between accepted and confirmed. Treating a timeout as failure is the mistake. A timeout means "I do not know." Every retry path for a side effect must start from that sentence, not from an assumption that nothing happened.

That distinction also explains why "just make the step idempotent" is not a complete answer. Idempotency is a property of the provider call, and it only helps if the retry sends the same key the first call used. A retry that generates a fresh random key on every attempt is not idempotent at all — it is a new request wearing a retry's clothes.

Build the reconciliation key first

The fix starts before the provider is ever called. Derive one stable key from the business event, not from the attempt. A good key answers: which workflow, which source record, which effect, and which version of the payload?

reconciliation_key = "n8n:notify:row-1042:send:v1"

Use the source record's stable identifier, never a run id or a timestamp, because a retry gets a new run id and a new timestamp by design. If the payload changes in a meaningful way — a different recipient, a different amount, a different message body — that is a different effect and it deserves a different key, or at least a version bump. If only the formatting changes, keep the key and let reconciliation treat it as the same effect.

Write that key to your own ledger before you call the provider, with status pending. A sheet row, a database table, or a key-value store all work; what matters is that the ledger survives the workflow run. If the workflow crashes, the ledger is the memory it wakes up with. The ledger row should hold the key, the source id, the recipient, a hash of the payload, the time of the first attempt, and later the provider's record id.

Then send the key with the request, in the provider's idempotency header or idempotency field when one exists. Many providers honour a key for 24 hours or more, and will return the original result instead of creating a second record. That alone fixes the duplicate in the common case — but only if you also handle the case where the provider has no such feature, or the first request never carried the key. Do not assume the provider saved you. Verify.

Reconcile before you resend

The retry branch must not go straight back to the send node. It must go to a check node first, in this order:

1. Check your ledger. If the key is already confirmed with a provider record id, stop. The effect happened. Mark the run complete and do not call the provider again.
2. Check the provider by key. Query the provider's list or search endpoint filtered by your reconciliation key, external id, or the recipient plus payload hash — whichever identifier the provider actually stores. If it returns exactly one matching record, write its id into your ledger, mark it confirmed, and stop.
3. Only then consider sending. If both checks return zero matches, and only if the provider confirms the key is unused, send once with the same key and immediately record the result.

That order matters. Ledger first is cheap and fast. Provider second is the authority, because the ledger can be stale if the first attempt wrote to the provider but crashed before updating the ledger. Sending last, and only after two negative checks, is what keeps a slow provider from turning one message into two.

Here is the shape of the check in a Code node, using a provider search that returns matching records:

// Input: reconciliation_key, and the provider search result in $json.matches
const matches = $('Provider Search').first().json.matches ?? [];

if (matches.length === 0) {
  return [{ json: { decision: 'SEND', reason: 'no provider record for this key' } }];
}
if (matches.length === 1) {
  return [{ json: {
    decision: 'SKIP',
    reason: 'provider already holds this effect',
    provider_id: matches[0].id
  } }];
}
// More than one match is not a retry case. It is a data problem.
return [{ json: {
  decision: 'REVIEW',
  reason: `provider holds ${matches.length} records for one key`,
  provider_ids: matches.map(m => m.id)
} }];

Notice what this code refuses to do: it never treats "search returned nothing" as proof that nothing happened, unless the search actually ran and the provider actually answered. A failed search, a search timeout, or a search that cannot filter by key must route to review, not to send. Otherwise you have rebuilt the original bug one node later.

Use the provider's own count as an assertion

An idempotency key is a request to the provider. A count assertion is a check on the result. Both belong in the workflow, because providers differ in how well they honour keys, how long they retain them, and whether they return the original record on a duplicate.

After any send — and after any skip decision — assert the count you expect. Query the provider for records matching your key and confirm the number is exactly one. If the send succeeded, the assertion is count === 1 and the returned record id matches your ledger. If you skipped because a record already existed, the assertion is the same: exactly one, and it is the record you found.

// Assertion after send or skip: exactly one provider record per key
const count = $('Provider Search').first().json.matches.length;
if (count !== 1) {
  throw new Error(`Reconciliation failed for key ${$json.reconciliation_key}: expected 1 provider record, found ${count}`);
}

Throwing here is deliberate. A count of zero after a supposedly successful send means the confirmation and the record disagree, and the workflow should not silently move on. A count above one means a duplicate already exists, and automatically "fixing" it by deleting a record could destroy the one the customer already saw. Both cases need a human to look at which record is real before anything else happens.

The same assertion protects the next run, not just this one. If a later workflow step reads "did we notify this customer?" from the ledger, it can trust the ledger only because the send step proved the provider count. Without that proof, the ledger is a hope, not a record.

When to stop automating and send it to review

Reconciliation has a boundary, and crossing it automatically is how a small duplicate becomes a larger incident. Route to human review — a quarantined row, a Slack message, or a held task, never a silent retry — in these cases:

• The provider returns more than one record for one key, or the same key with different payloads. Something already diverged; a machine cannot decide which copy the customer acted on.
• The provider search fails, times out, or cannot filter by your key. You cannot prove zero, so you cannot prove sending is safe.
• The ledger says pending and the first attempt is older than your review window — say, one hour for a notification, much shorter for a payment. A stale pending record may mean the provider accepted it long ago and your check is querying the wrong scope.
• The provider record exists but its amount, recipient, or content does not match your payload hash. The key collided, the payload changed, or the wrong record was returned. All three need a person.
• The provider offers no search, no idempotency key, and no record id at all. With no way to check, there is no safe automatic retry for a side effect. Send once, and if that send is unconfirmed, hold it for review rather than guessing.
• The effect is irreversible or expensive: a charge, a refund, a shipment, a legal notice. The cheaper the effect, the more automation you can tolerate. The more irreversible it is, the earlier review should start.

Review is not failure. It is the workflow admitting the exact limit of what it can prove. A queue of five held items that a person clears in ten minutes is a far better outcome than five hundred quietly duplicated messages that a person discovers from customers.

A checklist for your next side-effect node

Before you ship a workflow that calls the outside world, check these in order:

1. Does every side-effect call carry a stable key derived from the business event, not the run?
2. Is that key written to a durable ledger as pending before the call?
3. Does a retry go to a check branch — ledger, then provider — instead of straight back to send?
4. Does the workflow assert provider count === 1 after send or skip, and fail loudly otherwise?
5. Is there a named review path for every case the check cannot resolve, with the key, the payload hash, and the provider ids attached?
6. Have you tested the timeout case with a mock that accepts first and answers second, not just the clean-error case?

If any answer is no, the workflow does not have a retry policy. It has a duplicate policy that has not fired yet.

The rule I keep coming back to: a retry is safe only when the workflow can prove, from a record outside itself, that the first attempt did not already do the thing. Until it can prove that, the correct action is not to send again. It is to look.

Free workflows from this series (MIT):

https://github.com/zhp910318/n8n-ai-automation-lab

The production templates in my store:

https://payhip.com/haopengailab