Every weekday at 02:15 and again at 14:40, our address validation calls started returning 429 for about twenty minutes. Interactive checkout traffic failed, customers saw a generic error, and by the time anyone looked the window had closed. Our own request volume at those times was ordinary. We had a per-service client with sensible concurrency, and nothing in our metrics suggested we were anywhere near a ceiling.
We were not. The finance team was. Two years earlier somebody had signed up for the vendor once, taken an API key, and pasted it into a shared secret. That key was now in our checkout service, in a nightly reconciliation job, and in an analytics notebook that ran on a schedule. The vendor's limit was per account, not per key or per caller, so the reconciliation job burning through a hundred thousand addresses in a burst consumed the whole quota, and the customer-facing path lost the race because it was the smallest consumer.
Nobody was wrong locally. The batch job was doing its work as fast as it could, which is exactly what it had been asked to do. The vendor was enforcing a documented limit. Our monitoring was per service and could not see a quota that was shared across three systems and two departments.
The fix started with visibility. The vendor returns X-RateLimit-Remaining, and we had been discarding it. It is now a gauge, exported by every caller, tagged with the caller name, so one graph shows the whole account's headroom and who is eating it. Alerting on remaining headroom rather than on our own error rate would have caught this in the first week.
Then we split the accounts: interactive traffic has its own key and its own contractual limit, batch has another, and the notebook talks to a copy of the data instead of the live API. Anything non-interactive runs at a deliberate rate with a token bucket set well below its share, because finishing at 03:30 instead of 02:35 costs nothing.
A rate limit is a shared resource with no owner. If you cannot name every process holding your credentials, you do not know what your capacity is, and you will find out during checkout.
– Sergey Shinder
Top comments (0)