Our HTTP client timeout for the charge call was 10 seconds. The gateway's upstream timeout to the issuer was 30. Most auths come back in 1-2 seconds, so nobody noticed the gap.
Then one issuer's processor started taking 12-15 seconds on a bad afternoon. Our client gave up at 10, logged "payment failed", and showed the customer a retry button. The auth was still in flight and succeeded at second 13. The customer clicked retry, the new attempt got a fresh order ID, and we ended up with two approved authorizations on one cart. A cleanup job voided one of them hours later, but the cardholder's available balance stayed held twice for days.
The mistake was treating a timeout as a decline. A timeout is a third state: unknown. The fix was small:
- On timeout, mark the payment pending_unknown, not failed
- Poll the gateway by our merchant order ID for up to 60 seconds before allowing a new attempt
- Reuse the same order ID and idempotency key on retry, so the gateway returns the original result
- Set the client timeout above the gateway's upstream timeout, not below it
Duplicate holds on slow-issuer days dropped to zero over the next two weeks.
How do you handle the unknown state in your checkout? Do you block the retry button, or let the idempotency key sort it out?
Top comments (7)
The double-hold on the cardholder's balance is the detail that makes this worth writing down, because it turns an engineering mistake into a support ticket and sometimes a chargeback. We landed on the same third state eventually, but the hard part was not the code, it was the UI: 'pending_unknown' is an awkward thing to show a customer, and the temptation to render it as failed is exactly what produced the retry button in the first place. Setting the client timeout above the gateway's upstream timeout is the cheap half of the fix and worth doing even before any polling exists.
Exactly, the UI is where the bad decision gets made. Developers know “unknown” is a separate state, but the screen only has room for success or failure. What’s worked for us is not showing the customer the word “unknown” at all. Instead we show something they can act on: “We’re confirming your payment, this can take up to a minute. Please don’t pay again.” That covers the polling window, and only after it does the UI move to a real success or failure. And fully agree on the timeout order: it’s a one-line config change, and it removes most of the problem before any polling logic exists.
That copy is the right move. "Please don't pay again" is doing the work a decline would otherwise push onto support, and giving the customer an action beats a status word every time. The one thing I'd watch: "up to a minute" is now a promise, so generate it from the same config that sets the polling deadline rather than hardcoding it next to the UI. If someone later stretches the poller to 90 seconds, a hardcoded line becomes the new overpromise, and the customer pays twice on the strength of a string literal. Tie the two together and the screen can't drift from the backend.
Good catch, a string literal promising a timing is a bug waiting for its config change. Driving the copy from the same value as the poller deadline keeps them honest. The same applies to the other end of the window: if the deadline passes and the status is still unknown, the screen shouldn’t fall back to “failed”. Something like “We’re still waiting for your bank to confirm. We’ll email you as soon as we know, please don’t pay again” keeps the promise intact. Reconciliation then resolves the payment in the background.
That far end of the window is where most checkouts cheat. The poller gives up, the screen flips to "failed", and the bank later approves anyway. Two things I'd bolt onto your version: "we'll email you" moves the trust to the email channel, so that email needs the same idempotency discipline as the webhook handler, exactly once even when the confirmation arrives twice. And the "still waiting" screen needs an exit of its own, a reconciliation deadline after which it hands over to something a human can act on, otherwise the honest copy becomes an honest dead end. Your two comments here basically write the state machine: pending, confirming, still-waiting, resolved.
Your last fix is the one I'd underline: each side has to wait longer than the hop behind it. On a prepaid top-up platform I worked on, a request crossed two to four hops before reaching the carrier, and a terminal that gives up first retries while the original is still alive inside. Polling only worked one way: terminals could ask the platform about their own ID, but the carriers had no status query, so an unknown top-up stayed unknown until the next day's reconciliation file. To your question, I'd block the button: retry asks for the status first and, with no answer, shows "in progress, don't repeat it". The idempotency key is the safety net, not the plan. I wrote up the whole case here: dev.to/crtdrops/the-top-up-nobody-...
Great case, Israel, thanks for sharing the write-up. A hop with no status query is the worst version of this problem, because “unknown” can’t be resolved in real time at all, only through reconciliation the next day. Agreed on blocking the button: the retry should first ask for the status, and the idempotency key is the safety net for when that check fails or a second request slips through anyway. One thing we’d add for chains like yours: every layer that can’t query the hop behind it should still keep its own record of what it sent and when. Then at least the reconciliation file has something to match against.