Sam Altman says the RAM in your computer is a bad use of the world's memory. He's half right, and the half he skips is the reason a lot of our clients ask us about local models in the first place.
Here's what he said. On the Cheeky Pint podcast with Stripe co-founder John Collison, about 29 minutes in, Altman was asked about supply chain shortages. As quoted by The Verge on October 30, he said:
"A lot of people want local inference. And somehow, they assume that the problem is solved now because we are going to have open-source models, we're going to be able to run them on our computers, and so we're not going to need all the stuff in the data center. But from a global allocation perspective, actually, it's quite terrible to have everybody have a bunch of the world's limited supply of DRAM sitting in their own computer and being used five percent of the time. You want maximum utilization in the cloud if you're trying to deal with this very supply constrained environment."
Developers did not take it well. There's a busy Hacker News thread, r/LocalLLaMA lit up, and most of the reaction boils down to "the man who sells cloud inference says cloud inference is better." Fair enough. But I build AI features for clients for a living, and I think both camps are arguing past the decision teams actually face.
So here's my honest take: his arithmetic is mostly right, his framing is wrong, and neither one tells you where to run your model.
What's actually true in what he said
Start with the part people don't want to admit. Utilization matters a lot for cost.
A GPU or a block of memory in a data center serving thousands of requests is busy most of the time. The same hardware on a developer's desk is idle overnight, idle in meetings and idle at lunch. If you divide the hardware cost by the tokens it actually produces, a busy shared machine wins easily. That's not an OpenAI talking point; it's the same reason most companies stopped running their own mail servers.
And the memory market really is tight right now. TweakTown, reporting on the same comments, pointed out that consumer DDR5 kits have jumped sharply in price over the past year and cited TrendForce forecasting another 10% to 15% rise in conventional DRAM contract prices this quarter as suppliers prioritise AI server memory. The Verge noted that Samsung and SK Hynix have long-term deals with cloud operators and projects like OpenAI's Stargate that reserve much of their output for the next few years.
So if your only goal is "cheapest tokens for a general chatbot", the cloud usually wins. I've told clients exactly that.
What the framing leaves out
The problem is the premise that people run models locally to save memory, or to save money. In our work, they almost never do.
When a client asks me about local or self-hosted models, the reason is nearly always one of these:
- The data can't leave. Contracts, health data, internal financials, source code under a strict NDA. For some of these, "the vendor promises not to train on it" isn't enough for their compliance people.
- They want nobody else's policy in the loop. A hosted model can be deprecated, rate-limited, repriced, or start refusing a task your product depends on. A model file on your own hardware can't change under you.
- It has to work offline or at the edge. Field apps, factories, a laptop on a plane, a device with bad connectivity.
- The workload is steady and boring. Classifying documents all day, every day, at a predictable volume. Here owned hardware runs hot, and the idle-time argument mostly goes away.
None of those is an allocation question. Altman's quote treats local inference as a worse way to do the same thing. For these teams it's a different product: one where control and privacy are features, not inefficiencies.
There's also an irony nobody in the thread missed. The scarcity he describes is driven by AI data center demand, including his own company's. Asking consumers to give up local memory so the cloud can have more is a hard sell from the biggest buyer in the market. I don't think that makes him wrong about utilization. It does mean "global allocation" isn't a neutral frame coming from him.
How we actually decide: a routing table, not a religion
Here's the rule of thumb we use on client projects. It's not clever, but it's saved us from both extremes.
| Workload | Where it runs | Why |
|---|---|---|
| Frontier reasoning, long context, agents | Hosted API | Capability you can't match locally, bursty usage |
| Sensitive data, strict compliance | Self-hosted open-weight model (client's cloud or on-prem) | Data stays inside their boundary |
| High-volume, simple, predictable (classify, extract, tag) | Small self-hosted model or hosted small model, whichever is cheaper at real volume | Utilization decides; measure it |
| Offline or edge | On-device small model | No network, no choice |
| Prototype or MVP | Hosted API | Speed of iteration beats everything early on |
Notice the middle option that both sides of the debate skip: open-weight models running in a data center you control. You get most of the utilization benefit Altman describes, and most of the control the local crowd wants, because you can move the weights tomorrow. For a lot of B2B products that's the sweet spot.
Make the choice reversible in code
The bigger mistake I see isn't choosing the wrong side. It's hard-wiring the choice. If your Django views call one vendor SDK directly, switching later is a rewrite. A thin interface keeps it a config change:
# ai/router.py
from dataclasses import dataclass
from django.conf import settings
@dataclass
class Task:
name: str # "classify_invoice", "draft_reply", ...
sensitive: bool # contains data that must stay in our boundary
text: str
def pick_backend(task: Task) -> str:
if task.sensitive:
return "self_hosted" # e.g. an open-weight model behind an OpenAI-compatible server
return settings.AI_ROUTES.get(task.name, "hosted")
def complete(task: Task) -> str:
backend = pick_backend(task)
client = settings.AI_CLIENTS[backend] # both expose the same .complete()
return client.complete(task.text)
Many self-hosted inference servers expose an OpenAI-compatible endpoint, so the "client" for each backend is often the same SDK pointed at a different base URL with a different model name. Routing becomes a settings dictionary, and moving a task from hosted to self-hosted (or back) is something you can test with an eval set instead of debating in a meeting.
Two more habits that matter more than which side you pick:
- Log cost and latency per task, per backend. Then the "is local cheaper?" argument is a query, not an opinion.
- Keep a small eval set per task. Thirty to fifty real examples is enough to tell whether a smaller or self-hosted model is good enough for that job.
Where I've landed personally
I run small models on my own machine for experiments and for poking at client data I'd rather not send anywhere during early discovery. The RAM is already in the laptop. I'm not buying 128GB at today's prices to save on API bills, because at my personal volume the cloud is cheaper and I know it. When I need serious reasoning, I use a hosted frontier model and don't feel guilty about it.
For client products, I've stopped treating "local vs cloud" as a values question. It's a routing question, answered per task, and it should be cheap to change your mind. The teams that get burned are the ones who picked a side in a Twitter argument and then built their whole stack around it.
So yes, Sam, idle memory is wasteful. But for a lot of the teams I work with, the RAM sitting idle on their own hardware is what control costs, and many of them are happy to pay it.
What do you think?
Where do you draw the line: which AI tasks in your product would you never send to a hosted API, and which would you never bother running yourself?
Top comments (8)
One thing I'd add to the local inference discussion is that model weights aren't the whole memory budget. KV cache can become the actual bottleneck once you move from testing a model with one prompt to serving multiple users.
Take a self-hosted RAG service where several requests arrive with 32k-token contexts. Even if the quantized model weights fit comfortably in VRAM, concurrent sequences keep allocating KV cache as their contexts grow. Eventually, the scheduler has to preempt requests and recompute them, which can push tail latency up even though the model itself hasn't changed.
This is where vLLM's PagedAttention and prefix caching become interesting. They help manage KV memory and reuse shared prefixes, but they don't make the capacity problem disappear. I'd benchmark with realistic context lengths and concurrency, then watch KV cache utilization, preemption counts and p95 latency rather than just checking whether the model loads successfully.
A local model that fits isn't necessarily a local inference service that scales.
Great point about KV cache. We've seen the same thing: a model that loads fine in a single-prompt test can fall over once several long-context requests run concurrently. Tracking KV utilization, preemption counts and p95 latency under realistic concurrency is a much better readiness check than "it fits in VRAM", and we should have called that out in the post.
Exactly. I'd also look at how the scheduler handles mixed workloads, not just concurrent requests with similar context lengths.
For example, in a RAG service, one request might carry a 2k-token context while another needs 32k. If long-running sequences keep occupying KV blocks, shorter requests can end up waiting behind them even when their actual generation workload is relatively small.
I'd test different batching and scheduling policies under that kind of workload, especially chunked prefill and prefill/decode disaggregation. The interesting part is whether they improve time-to-first-token without pushing inter-token latency in the wrong direction.
Average throughput can look healthy while individual requests are having a pretty bad time.
That's a useful distinction: average throughput can hide head-of-line blocking when a 32k-token request sits next to a 2k one. We'd want to measure time-to-first-token per context-length bucket, so chunked prefill gets judged on the short requests it is meant to help. Splitting prefill and decode is on our test list for the same reason, with inter-token latency tracked alongside.
The real issue is latency. That's the true cost of local memory. Sounds counter intuitive, but 128gb of memory in a server, means more than 128gb of system memory at home, spread across 4 computers. It can be shared, sharded, repurposed, etc. Without ever being a problem to manage and it'll never go to waste. But that's only half the equation. What do you need memory for? To store game-states? To run a local LLM? Yet, you use it at most for 16 hours in a day actively and you will hardly ever actually utilize 100% of it. That's wastage. Especially when you consider in today's life, what you're actually using it for, is a local state clone of what's already running in the cloud? Whether it's viewing a website, local IDE state, a multiplayer game, etc. It's a waste of potential and often times, outright inefficient.
Take an online multiplayer game. CoD for instance, you need a pretty hefty pc to run it? What if it ran on the server and all your pc did was do a hash and render your game-state from that hash? You could run it at max detail on an iGPU, with next to no memory usage, if it were done properly. The cloud can run the actual game, with fragmented client states all in memory on the server. What that would mean, there's no 'lag' for inputs, only render latency. The server could with 100% certainty say who clicked first. No race conditions at all. It'd take a fraction of the compute and the only drawdown would be slightly higher network usage.
So yes, it's a waste having system memory, it wastes potential and it's built that way, because of an inefficient system.
Thanks for the thoughtful take. We agree that pooled, shared memory is usually used more efficiently than idle RAM spread across several home machines. Our "half right" view is that latency, privacy and offline work still push some client workloads local, so for us it comes down to what each workload actually needs. Streaming a rendered game state is an interesting idea, and the trade-off would be round-trip latency and network dependence.
True, but how much of that is due to existing paradigms and lack of innovation? Windows demands a minimum of 8gb of ram to run smoothly, but that's due to the local processing most things rely on, yet majority of people only use a computer for things they do online (or could do online). With the round-trip latency it's actually a catch-22, because currently, the same latency exists, except you get inconsistent behavior, because what your screen shows vs what the server ran, are off by eg. 200ms, that's probably the nr 1 reason people scream at their screens playing CSGO, because 64 tick servers are iffy at best determining who shot first and due to the cross-device latency, when you look at a replay with debug on, you'll see that what was on-target on your screen, was registered as off-target on the server. This would solve that permanently. Yes, network dependency would become a thing, but arent we already network dependent? Except with this system, you'd be able to run the latest AAA game on a potato pc, which opens up a much broader market and makes it more accessible. Same goes for CAD, no more need to have 64gb of ram to run a model, you run in cloud and it auto-scales?
There will always be tradeoffs, but imo the benefit far outweigh the drawdowns, take privacy and latency for instance, with a simple example, you're running an AI agent, in cloud, in a wasm process, that would be completely sandboxed and disposed of on completion, with near 0 latency execution time due to server proximity, your IDE would just render a canvas, as opposed to actually running anything. That speeds up your loop and you get more work done, equally as privately.
The only scenario where local memory is actually necessary, is offline work, which is a rare occurrence in 2026?
Fair challenge. Part of it is certainly habit: a lot of what runs locally is just inertia, and for browser-first work a thin client plus a fast connection is often enough. Where we still see local memory win is when the round trip is the bottleneck, such as interactive tools, large local files or offline use. So we'd treat it as a per-workload call rather than a rule either way.