Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run
A webhook arrives: order.updated, subscription.updated, issue.edited. The body contains the object, or most of it. Your handler has two choices:
- Trust the payload. Read the new state from the event body and write it to your database.
- Treat the event as a doorbell. Note the object's ID, call the sender's API, fetch the current state, and write that.
Both are common. Both are recommended by someone. Some providers have started sending "thin" events that contain only an ID and a type, which makes the choice for you. I think this is one of the more interesting design arguments in integrations, so let's have it.
The case for trusting the payload
It's fast and cheap. No extra API call, no extra latency, no extra point of failure. For a high-volume integration, re-fetching every object can double your API usage or more, and push you into the sender's rate limits exactly when traffic spikes.
The sender's API might be down. Webhooks are often queued and retried by the sender for hours or days. If the sender's API has an outage, a fetch-first handler can't process anything, while a payload-first handler keeps working from the events it already has.
The payload is what the sender meant at that moment. For event-like facts ("payment succeeded at 12:01", "user clicked unsubscribe"), the payload is the record. Fetching later may return a state that has since moved on, and you lose the history of what happened.
It's what most docs show. Tutorials, SDK examples and quick-start guides overwhelmingly read from the event body. Code that follows the docs is easier for the next developer to understand.
The case for fetching the latest state
Webhook order isn't guaranteed. Most providers say explicitly that events can arrive out of order and more than once. If you apply payloads as they arrive, an older event processed late overwrites newer state. Fetching makes order irrelevant: every handler converges on the current state.
Payloads can be stale even when they arrive in order. Retries deliver the original payload, which may be hours old. A handler that applies a three-hour-old "status: past_due" to a subscription that has since been paid is doing damage.
Payloads may be incomplete. Many APIs send a partial object, omit expanded relations, or truncate large fields. Handlers that trust the payload start growing special cases ("if the field is missing, fetch it anyway").
It simplifies the code. Instead of one handler per event type, each with its own idea of what changed, you have one "sync this object" function, called by every event, by a nightly reconciler, and by an admin button. Three triggers, one code path.
It's more secure against forged events. If someone bypasses your signature check, a fetch-first handler still only writes what the real API says.
Where I've landed
My default is: fetch for state, trust for facts.
- State is anything that can change again: subscription status, order status, user profile, issue labels. For these, the payload is a hint. Fetch the current object and write that. Out-of-order and duplicate events become harmless.
- Facts are things that happened once and won't un-happen: a payment captured, a message delivered, a form submitted. For these, the payload is the record. Store it as an event, idempotently, keyed by the event ID.
Then three supporting habits:
- Debounce bursts per object. If five events arrive for the same subscription within a second, fetch once. A short delay ("sync this object in 2 seconds, collapsing duplicates") cuts API calls dramatically. This is a good use of delayed jobs, whether through something like delay.run or a queue with deduplication keys.
- Fall back to the payload when the API is unavailable, but mark the record as "synced from event" and re-sync later.
- Reconcile on a schedule. Whatever you choose, a periodic job comparing your state with the provider's catches what both approaches miss.
Need a second look at the event, a few seconds later? delay.run schedules a delayed HTTP callback with retries, so you can fetch the latest state after the webhook arrives.
The cases that start arguments
- Thin events. If the provider only sends IDs, you have to fetch. Is that a better design, or just shifting cost to the receiver?
- Version numbers. Some APIs include a monotonically increasing version or sequence number on each object. With that, you can trust payloads safely by ignoring anything older than what you have. Is that enough to make fetching unnecessary?
- Rate limits. At what volume does fetching stop being reasonable? Hundreds of events a minute? Thousands?
- Your own webhooks. If you send webhooks, should you send full objects or thin events with a fetch URL? What do your consumers prefer?
What the hybrid looks like in code
A minimal version of "fetch for state, trust for facts", in pseudo-Node:
async function onEvent(event) {
await db.events.insertIgnore({ id: event.id, type: event.type, payload: event });
if (FACT_TYPES.has(event.type)) return recordFact(event); // payment captured, message delivered
const id = event.data.object.id;
await jobs.enqueue("sync-object", { id }, { delaySeconds: 2, dedupeKey: `sync:${id}` });
}
The fact path stores what happened, once. The state path collapses a burst of events into one delayed sync job per object, which then calls the API and writes the current state. Both paths are idempotent, and neither cares what order events arrived in.
A quick test for your own handler
Take one handler and ask:
- If this event arrived twice, would anything break?
- If it arrived after a newer event for the same object, would it overwrite newer data?
- If it arrived three hours late, would it apply old state?
If any answer is "yes", the payload-trusting version needs either version checks or a fetch.
Related reading
- Case study (hypothetical): the customer who paid, but the app still said 'Free plan'
- The transactional outbox, explained: how to save to the database and enqueue a job without losing either
- Exactly-once scheduled jobs don't exist: design for at-least-once and stop double-charging
- Case study (hypothetical): the payment webhook that arrived before the order existed
Which way do your webhook handlers lean: payload or fetch? And if you trust the payload, how do you deal with events that arrive out of order?
Top comments (7)
The useful split is not trust versus verify, it is what the handler needs: the delta, or the state. If the event carries a version, you can apply deltas safely by keeping the last applied version per object - idempotent, ordered, no extra read. If it does not, you are guessing about ordering anyway, so the fetch buys correctness you cannot get another way.
The cost of the fetch is easy to underestimate: one extra read per event, so a busy endpoint doubles its read QPS on that table at exactly the moment the write burst is happening.
Cheapest middle ground: fetch only on the event types that touch state you cannot reconstruct from the payload.
A practical split is to use the provider’s object version in a conditional upsert for mutable state, ignoring events older than the stored version; event IDs prevent duplicates but not reordering. If there’s no reliable version, store each event idempotently and coalesce a fetch by object ID. Keep one-time facts in an append-only event record so a later fetch can’t erase them.
One addition: "event IDs prevent duplicates but not reordering" is the sentence I would put at the top of the design doc, because most implementations quietly treat dedupe as ordering.
The conditional upsert on the provider's version is what makes it actually safe, and the append-only side record is what stops a later fetch from rewriting history - that is the failure that only surfaces in an audit months later, when the one-time fact has been overwritten by a state snapshot that looked more recent.
If you wait and fetch once for several events, then fall back to the payload when the API is down, the single write can be stale. It is easy to miss because the debounce worked and the fallback worked, yet together they write old state. Anyone hit that in practice? Disclosure, I built Webhook Relay. It can store and retry incoming deliveries, but that does not fix a stale payload after an API outage.
Yes, and the reason it is easy to miss is that both mechanisms are individually correct: the debounce coalesces events, the fallback keeps the write alive. They are jointly unsound because they read from different sources of truth at different times.
The fix I would reach for is making the version gate the only path to a write: the fallback payload still carries the event's version, so the same compare-and-apply runs whether the value came from the API or the body. A stale payload then fails the gate like any other old event, instead of being trusted because it arrived through the resilient branch.
Retrying deliveries helps the lost case, not the stale one - you are right that they are different problems.
Team doorbell, with one amendment: fetch the latest state, but keep the raw payload too. Re-fetching wins on ordering — a stale event can't overwrite newer state — but the payload is your evidence of what the sender believed at send time, and when the two disagree that difference is the bug report. In document-exchange work we keep both: the fetched state drives processing, the original bytes drive the argument about who changed what, when.
Agreed, and I would make the storage shape part of the rule rather than a convention: keep the raw bytes with the received timestamp and the event id, immutable, next to the version you actually applied. Then "who changed what, when" is a query instead of a memory.
The metric that comes free is the disagreement rate - payload version versus fetched version at the same event id. It should sit near zero. A nonzero rate is either clock skew on the sender side or a sender that serialises state after emitting the event, and both are worth knowing before they become a dispute.