We deduplicated webhooks the obvious way. Handler receives an event, runs SELECT on processed_events by event_id, and if nothing is found, does the work and inserts the row at the end.
It held for months. Then a customer got two receipt emails and a loyalty-points credit twice for the same payment.
The cause: the gateway redelivered the event 40 ms after the first attempt, because our first response was slow (a cold DB connection pushed it to about 3 seconds, and their timeout was 2). Both deliveries landed on different pods. Both ran the SELECT before either had inserted. Both saw an empty table, and both did the work.
In logs it looked like duplicate delivery, but the gateway was behaving correctly. Our check-then-act was the bug.
The fix was to flip the order. Insert the event_id first, into a table with a unique constraint, inside the same transaction as the side effects. If the insert hits a conflict, return 200 and stop. If the transaction fails, the insert rolls back and the next redelivery gets a clean retry.
Two details that mattered. We key on the event id, not the payment id, since one payment emits several events. And we return 200 on conflict, because a 4xx there makes the gateway keep retrying a event you've already handled.
How do you handle side effects that can't live in the same transaction, like email sends? Outbox table, or something else?
Payneteasy handles this kind of payment flow for merchants and PSPs — see how it works.
Top comments (1)
Outbox for the email, and the sender checks a sent marker keyed by the same event id, so a second send is rare and harmless when it happens.
The harder case is a side effect that moves money somewhere else, like a top-up sent to a carrier. It can't live in your transaction and it can't be undone, so the outbox only gets you "at least once", and "at least once" is two top-ups. What worked for us: our own external id sent with the request, so a retry can be recognized on the other side, and a record that stays "in process" when the carrier doesn't answer, instead of being retried blindly. When the carrier has no status query, the daily reconciliation file is the only thing that closes it.