If you are building autonomous AI agents, you eventually hit a scaling wall. While developing Saturn AI, I noticed that pushing past five or six simultaneous agents caused the entire X11 Docker container to choke, API requests to time out, and the system to crash.
The issue was not the LLM API latency. It was OS process thrashing.
The Root Cause: Process Overhead and Disk I/O
When prototyping, it is common to rely on CLI wrappers to orchestrate agents. However, this introduces massive overhead. If six agents take a turn simultaneously, the backend spawns six heavy Node or Rust processes. If those agents invoke tools, they spawn additional child processes.
The container rapidly runs out of memory, IPC pipe bandwidth, and CPU threads. Furthermore, if these wrappers maintain state by constantly reading and writing JSON session files, the disk I/O locks up completely.
Three Architectural Fixes for Multi-Agent Stability
To resolve this and scale efficiently, you have to treat agent turns like asynchronous network requests rather than OS shell processes.
1. Implement Concurrency Queuing
If you cannot rewrite your engine immediately, introduce an asynchronous job queue using a library like p-queue. Cap the concurrency to two or three active processes at a time. When a trigger wakes up six agents, the queue allows the first few to execute while the others wait in memory. This eliminates CPU thrashing and keeps response times stable.
2. Shift to In-Process SDK Calls
The long-term fix is removing the CLI middleman entirely. Build a custom ReAct loop using a native framework directly inside your main event loop. By executing agent turns as standard asynchronous network calls to the LLM provider, you can run dozens of concurrent agents in a single instance without spawning external processes.
3. Use a Shared "Blackboard" Memory Model
Isolated JSON files for state management will bottleneck your disk. Transition to a shared state model stored directly in memory or a local Redis instance. All agents can instantly read and write their context, tasks, and tool outputs from this shared space. Additionally, boot a single persistent tool server on startup, and have all agents route through it via internal WebSockets, rather than each agent booting its own tool instances.
By shifting away from process-heavy wrappers toward lightweight, async architecture, you can scale multi-agent environments reliably without burning through compute resources.
Top comments (8)
"The issue was not the LLM API latency, it was OS process thrashing" is the key
insight, and JSON session-file locking has killed more agent systems than any
model behaviour. p-queue first is the right order. The other axis, for when
the work genuinely is parallel: one machine per agent, billed by the minute so
idle costs nothing. At five or six agents a queue is clearly better though.
At what concurrency did isolation start paying for itself?
Thanks Dhruv! Glad that insight resonated. You're completely right about JSON session-file locking—it becomes an absolute bottleneck the second multiple agents try to read/write state simultaneously.
To answer your question: since I’m managing the execution sandboxes using Docker containers on fly.io, the isolation really started paying for itself around 3-4 concurrent agents.
Below that threshold, a standard queue handled the load perfectly fine without the extra overhead of spinning up separate isolated environments. But once I pushed past that, the local I/O thrashing was too heavy, and true isolation became mandatory to prevent race conditions and keep the environment stable.
Have you experimented much with different state management databases when scaling up to one machine per agent?
Fly Machines are already Firecracker microVMs, each with its own kernel. So if you're running several Docker containers inside one machine, they're still sharing that machine's kernel , and the fix might just be one machine per agent rather than one container per agent. The boundary you want is already there.
On state: one-machine-per-agent often removes the need for a state database entirely. The JSON locking was a shared-filesystem problem, not a data-modelling one. What replaces it is persistence across disposable lifetimes , snapshot the disk, or take an external DB and its coordination point back on.
Yes, I am building Saturn with one machine per agent.
That's the right shape. The next thing that usually bites is boot cost , one machine per agent means one cold start per agent, and re-running installs on the way up is where the latency goes. Booting from a prepared snapshot instead of provisioning fresh tends to be the fix.
Did you go snapshot-restore, or does each agent build its own environment?
Not quite. Fly.io is built to handdle these types of things. It noots up rrally fast too. The feature I use is where I dockerize my backend agent put it on ghcr the fly.io straight up taes my pre compiled image and runs it. The latency is very low. It gives a dynamic URL for each user and it provides a full REST API to automate the process. So this is ideal for isolated AI agents
Fair , a precompiled image is the same fix at a different layer, and if your agents start clean each time there's nothing left to optimise. The one place it diverges is resume: an image carries the toolchain, not the state. If yours are stateless per run, you're already at the right answer.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.