Rate Limits Are a Queueing Problem: Capacity Math for Agent Fleets
Standup ends. Five people go back to their desks and invoke five agents within about ninety seconds, because standup is what schedules agent work now, not anybody's planning.
By 10:30 the first person is staring at a spinner. By noon, the requests that should have finished by lunch are queued behind each other, somebody is complaining in the channel about "the limit hitting early today," and the provider is getting blamed for a traffic jam the team built themselves.
The provider is innocent. The math will convict the wrong defendant every time until you run it once.
Two Numbers and One Law
A queue has two rates. The arrival rate, how many units of work show up per unit of time. The service rate, how many units the server finishes per unit of time. Little's Law connects them to the backlog: the average number of items in the system equals the arrival rate times the average time each item spends in it.
One implication does all the work here: if arrivals exceed service over any sustained window, the queue grows without bound, and your latency stops being a property of your system and starts being a property of your backlog.
Your provider plan is the server. Its published limits, the tokens per minute, the requests per minute, the five-hour window, are the service rate. Your agents are the arrival rate. Nobody in the standup scene above ran these two numbers, which is why the spinner felt like bad luck instead of arithmetic.
The Herd, Not the Limit
Here is the part that took me too long to see. Agent arrivals are not random. They are scheduled by human rituals.
Standup ends, five agents fire. After lunch, the batch jobs wake up. Sprint planning ends, the spikes start. The arrival rate spikes on a calendar you can read, and the plan's service rate is flat all day. The gap between a spiky lambda and a flat mu is the entire complaint.
This is why the upgrade instinct is usually wrong. A bigger plan raises the service rate. The herd is an arrival-rate problem. When the spike is five deep against a plan that serves three, the fix that costs nothing is to stop releasing all five at 9:02.
tip: Before upgrading the plan, compute both sides. If arrivals only exceed service during predictable bursts, you have a scheduling problem, and scheduling is free. If arrivals exceed service all day, you have a capacity problem, and now the upgrade is the right conversation.
Three Levers, One of Them Costs Money
Batch. Hold arrivals in your own queue and release them at a rate you choose. A twenty-line drip feeder turns the 9:02 herd into a steady stream. Same work, same plan, no spinner at 10:30. This is the free lever and it goes first.
Checkpoint. Make interruptions survivable. A run that dies at the wall should resume from where it stopped, not restart from zero, because restarted work re-buys what you already paid for. This one is sneaky: a wall causes retries, retries inflate arrivals, and the limit feeds itself. Checkpointing breaks the loop.
Degrade. Ladder down mid-run. Small model for the routine steps, big model for the decisions, defer the optional work until the window resets. This lever needs a map of which steps are routine, which you should have anyway.
The fourth lever is the upgrade, and it is fine, but it belongs after the free three, not before them.
The Arithmetic in One Worked Example
Symbolic, so it survives the next pricing change.
A plan that sustains three concurrent heavy tasks. A team that fires five at standup plus three more drifting in over the hour. Eight arrivals in the 9-to-10 window against a server that finishes three. Five waiting, residence time climbing all morning, and the "limit" feeling personal by 11.
Apply batching: hold all arrivals, release three at a time, everything else waits in your queue where it costs nothing. Same eight tasks, same plan. Finish time moves from "never, effectively" to "early afternoon," and the only thing built was a drip feeder.
That is the entire argument for running the math instead of reading the dashboard. The queue was never going to negotiate.
The Honest Limitation
Agent arrivals are bursty, not Poisson, and the one-line Little's Law assumes averages. Real fleets have correlated bursts, retry storms, and the pathology where every agent's context grows through the day, so the afternoon tasks cost more tokens than the morning ones did.
The math gives planning bounds, not guarantees. It tells you which regime you are in: stable, borderline, or saturated. It does not tell you your exact 3 PM latency. Use it to pick levers, then watch the logs, because the logs are the measurement and the math is just the map.
Run It This Week
One week of usage logs. Total tokens over working hours gives your arrival rate. Your plan's published sustained rate is the service rate. Compare, and note the windows where the herd gathers. Then pick levers in order: batch, checkpoint, degrade, upgrade.
So here is the question, and it is answerable with a calculator and ten minutes: how many concurrent agents can your current plan sustainably run before somebody's five-hour wall hits at 2 PM, and do you know that number, or does the provider know it better than you do?
If the provider knows your capacity better than you do, you have been reading the wrong dashboard.
Top comments (5)
Framing the standup burst as the arrival rate is the useful shift, because the daily average can sit well under the limit while a 90 second spike blows past it. Have you compared staggering agent starts with a small random delay against a shared queue with a concurrency cap? I'd guess the cap is easier to reason about, since the five hour window then becomes a budget the team can plan against.
Yes, we see the bursts. Let me explain a bit - our agents run certain workloads and this involves EC2. The Linux EC2, no issues because aws scales well dynamically. However, our Mac runners we don’t. AWS charges for Mac runner for 24hrs regardless of usage, so the runner pool is fixed. Since runner pool is fixed, this is where the burst of agents hurt as now many agents are competing for resources where pool is fixed. As we are in 4 different time zones, people have adapted to kicking off their jobs as per TZ. Although standup happens at a fixed time for all TZs with best overlap. We encouraged folks not to kick off agents after standup as that’s when bursts happen. It’s not the best solution but it’s working with the constraints we have.
The hidden multiplier on the retry storm is that provider rate limits track rolling tokens rather than raw requests. When five agents get 429'd at the same second with twenty thousand tokens in flight, standard exponential backoff with full jitter still tends to synchronize. They all wake up right as the sixty-second token bucket refills, dump one hundred thousand tokens into the API at once, and knock each other back into rate limiting.
In my own multi-agent setups, what finally flattened the curve was decoupling tool execution time from LLM turns through a shared local leaky bucket. When one agent spends forty seconds waiting on a test suite or git clone, its slot on the token regulator opens up for another worker. Tracking token grants at the transport layer before the HTTP call ever fires keeps local concurrency high without ever triggering the provider backoff loop.
You describe a really interesting scenario where exponential back off is a great idea but given same software logic is running and the burst of retries keep happening at specific intervals although the cohort count may change. I know one may suggest a complex coordinator etc to allocate “come back at this time”. I dealt with this a decade or so ago with constraints on some resource (pre AI) with a back off where the retry was a randomized number between two limits. It was not perfect but the bursts of retries coming back at the same time with lull in between went down. Quite cheap to implement and it worked on production workload. To be precise, this was for replaying dead letter queue on a Microsoft Exchange server where emails were not really real time like a slack message. I got away with that design. The bursts were constrained by the Netapp storage IO which is where the Exchange dumped the dead letter queue emails. Will this work do you think?
The ten-fold token gap between the council and the critique subagent explains why in-context debates rot.
When an LLM writes both sides of an eight-persona debate in a single prompt, it spends tokens simulating conversational tension. The personas disagree on phrasing while sharing the same generation priors and context assumptions. None of them inspect the filesystem or test whether a path exists.
The critique subagent ran heavier because fresh context with an adversarial brief spent those 130k tokens exploring system state and running shell checks. Paying for an independent check that drops the wrong rate from 31 percent to zero is a trade I would take on any production migration.