Case study (hypothetical): the payment webhook that arrived before the order existed

This is a hypothetical case study, assembled from a pattern that shows up in a lot of payment integrations. The shop, the numbers and the timeline are invented; the race condition is very real.

The setup

A small online shop sells physical goods. Checkout works like this:

1. The browser submits the cart to the shop's backend.
2. The backend creates a payment intent with the payment provider and gets back an ID.
3. The backend saves an orders row with status awaiting_payment and the payment intent ID.
4. The customer completes payment on the provider's page.
5. The provider sends a payment.succeeded webhook. The shop's handler looks up the order by payment intent ID and marks it paid, which triggers fulfilment.

It worked fine in testing. In production, roughly 1 in 400 orders stayed awaiting_payment forever, even though the customer had been charged. Support found out when customers emailed asking where their parcel was.

What actually happened

Steps 2 and 3 were in the wrong order for a world with real latency, and step 5 assumed step 3 had already happened.

The backend created the payment intent first, then opened a database transaction to save the order. Under normal load the transaction committed in a few milliseconds. But with wallet payments, some customers completed payment almost instantly, and the provider is fast at sending webhooks. Occasionally, especially when the database was busy, the sequence went:

• 12:00:00.000 backend creates payment intent pi_123
• 12:00:00.180 customer pays with a saved wallet
• 12:00:00.240 provider sends payment.succeeded for pi_123
• 12:00:00.250 webhook handler: SELECT ... WHERE payment_intent_id = 'pi_123' returns nothing
• 12:00:00.300 checkout transaction commits the order row

The handler did what a lot of handlers do when they can't find the record: logged a warning and returned 200 OK. From the provider's point of view, the webhook was delivered successfully. It would never retry. The order sat there unpaid.

Why the obvious fixes weren't enough

"Just create the order first." Moving the insert before creating the payment intent helped a lot, and it's the right order. But it didn't remove the race completely: the order had to be updated with the intent ID afterwards, and that update could still lose to the webhook. It also meant orders existed for checkouts that failed before a payment intent was created, which needed cleanup.

"Return 500 when the order isn't found, so the provider retries." Better, and it would have fixed most cases. But the provider's retry schedule started at around a minute and backed off over days. A customer who paid at 12:00 and got a confirmation email at 12:01 is fine; a webhook that's retried for three days because of a real bug, filling alerts the whole time, is less fine. And a 500 for "not yet" looks the same in dashboards as a 500 for "crashed".

"Sleep for a second and look again." Inside the webhook handler, this holds a request open, slows every webhook, and still loses if the commit takes longer than the sleep.

The fix: accept, record, reconcile

The shop ended up with three changes, each small.

1. Store every webhook before doing anything with it

The handler now verifies the signature, writes the raw event into a webhook_events table with the provider's event ID as a unique key, and returns 200 immediately. Processing happens in a background job.

CREATE TABLE webhook_events (
  event_id      text PRIMARY KEY,
  type          text NOT NULL,
  payload       jsonb NOT NULL,
  received_at   timestamptz NOT NULL DEFAULT now(),
  processed_at  timestamptz,
  attempts      int NOT NULL DEFAULT 0
);

The unique key makes duplicate deliveries harmless: an INSERT ... ON CONFLICT DO NOTHING means the provider can send the same event five times and it's stored once.

2. Process with retries the shop controls

The job looks up the order. If it's found, it marks it paid in a transaction that also checks the current status, so running twice changes nothing:

UPDATE orders
SET status = 'paid', paid_at = now()
WHERE payment_intent_id = $1 AND status = 'awaiting_payment';

If the order isn't found, the job doesn't fail and doesn't give up. It schedules itself again: after 5 seconds, then 30, then 2 minutes, then 10. After about an hour without a matching order, it raises an alert, because at that point it's a real bug, not a race.

That schedule was the key difference. The shop no longer depended on the provider's retry timing; it decided for itself how long "not yet" was allowed to last.

3. Reconcile from the other side

Webhooks are a notification, not a source of truth. A scheduled job now runs every 15 minutes and asks the provider about any order that has been awaiting_payment for more than 10 minutes. If the provider says the intent succeeded, the job marks the order paid and logs that the webhook path missed it. This catches anything the first two parts don't: a webhook that never arrived, a signing secret rotated without updating the handler, a deploy that briefly broke the endpoint.

In the first month, the reconciler fixed 3 orders. Each one pointed to a different small bug, and each one was found within 15 minutes instead of by a customer email.

What I'd take from it

• Never return 200 for "I couldn't do anything with this" unless you've stored it. Accepting and dropping is the worst combination.
• Assume webhooks can arrive before, after, or instead of your own write. Ordering between your database commit and someone else's HTTP request isn't something you can rely on.
• Own your retry schedule. The sender's retries exist for when you're down; they're a poor tool for "I'm up but not ready".
• Reconcile. Any flow where money moves should have a periodic job that compares your state with the provider's.

Scheduling those short "try again in 5 seconds, then 30" follow-ups reliably is the kind of delayed job delay.run is designed for, but the pattern works with any durable job queue.

Related reading

• Return 200 fast: why your webhook handler should do almost nothing
• Make scheduled job handlers idempotent: safe to run twice
• The transactional outbox, explained: how to save to the database and enqueue a job without losing either
• Should cron schedules live in your repo or in your platform's dashboard? Let's argue about it

Have you hit this race in a payment or webhook integration? Did you fix it with retries, reconciliation, or by changing the order of writes?