Walk through an AI cluster and count the memory that is doing nothing. In a typical rack, between a third and a half of installed DRAM sits attached to a node that is idle, waiting for work, while a neighbouring node is starved and swapping.
This is not a provisioning mistake. It is what happens when memory is welded to a CPU socket and work is not — the capacity exists, and the topology forbids using it.
Part One: Why memory gets stranded
A server is sized for its peak workload. If one job needs 1 TB of RAM, every node in the pool gets 1 TB, because the scheduler cannot know in advance which node will receive the job. The moment jobs are heterogeneous — some memory-hungry, some compute-hungry — the average utilisation falls.
Cloud providers have lived with this for a decade and solved it with overcommit and migration. Migration is expensive: moving 1 TB across a network takes minutes and interrupts the job. Overcommit works only when the memory is not actually touched, which is not true for in-memory databases or large-model inference.
So the industry has converged on a different answer: stop attaching memory to a specific node. Put it behind a fabric, and let any node borrow what it needs.
Part Two: The latency tax, honestly
CXL memory is not as fast as local memory. Local DDR5 sits roughly 80 to 100 nanoseconds from the core. CXL-attached memory on a healthy fabric lands somewhere around 170 to 250 nanoseconds, depending on the switch topology and the generation.
That sounds like a rounding error until you count the accesses. A memory-bound workload issuing a hundred million loads per second at an extra 150 nanoseconds is not a small penalty — it is a structural one, because latency is on the critical path when the pipeline stalls.
This is why pooled memory is adopted first for capacity-tier workloads and last for latency-critical ones. A recommender's embedding table tolerates 200 nanoseconds. A database's hot index does not. The right design is tiered: hot pages local, cold pages pooled, and the operating system or the runtime decides which is which.
Pooling is not a faster memory technology; it is a cheaper way to own capacity you only occasionally need, and it only pays when the workload can tolerate the extra hundred nanoseconds.
Part Three: The arithmetic that decides it
Take a rack of eight nodes, each with 1 TB of DRAM, average utilisation 52 percent. You are paying for 8 TB and effectively using 4.2 TB.
Now add a pooling appliance with 2 TB of shared capacity, and reduce each node to 512 GB local. Total installed: 4 TB local plus 2 TB pooled equals 6 TB, a 25 percent cut in purchased capacity, with the ability to burst any single node to 2.5 TB.
The purchase cost falls. The interesting part is what happens to effective cost, which is purchase price divided by usable capacity. Local memory at 52 percent utilisation has an effective cost roughly twice its sticker price. Pooled memory at 85 percent utilisation has an effective cost close to sticker. The sticker comparison is misleading; the utilisation-adjusted comparison is not.
There is a second-order effect that matters more than the capex: scheduling flexibility. With pooling, a memory-hungry job no longer needs a specially provisioned node class. It takes what it needs from the pool and releases it. That removes an entire category of queueing delay, which in a cluster running expensive accelerators is often worth more than the memory saved.
Part Four: Where it breaks
Three failure modes are worth designing around.
The fabric becomes a single point of failure. A pooled memory appliance that serves sixteen nodes is, by construction, a shared dependency. Redundancy is mandatory, and redundancy eats the savings.
Bandwidth, not latency, becomes the limit. CXL links carry a fraction of what a local memory bus carries. A workload that streams data — not random access, but sequential scan — will saturate the link before it notices the latency. Pooling helps capacity-bound jobs and hurts bandwidth-bound ones.
The software does not know the topology. If the operating system treats pooled memory as ordinary memory, it will happily page a hot working set onto the far side of the fabric and destroy performance. Tiering has to be explicit, either in the allocator or in the application.
What is actually transferable
The deeper pattern is that every computing resource eventually detaches from the thing it was originally bolted to, and the detachment is always justified by utilisation rather than speed. Storage detached from the server and became a network service. Networking detached from the host and became a fabric. Memory is following the same path, and for the same reason: the average utilisation of a resource welded to a fixed unit is always poor.
The practical advice for anyone sizing a cluster in 2026 is to compute utilisation-adjusted cost, not list cost, and to check whether the workload is latency-bound or bandwidth-bound before assuming pooling helps. Most teams discover which one they are only after the first benchmark, and by then the purchase order is signed.
Pooled memory will not replace local DIMMs. It will absorb the capacity tier, the burst tier, and the multi-tenant tier, and leave local memory to do what it is genuinely best at: serving the hot path at full speed.
SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Creating this takes real effort — thank you for the support.
Top comments (0)