Analysis

Sam Altman on Local Inference: Right Math, Wrong Lever

Sam Altman says local inference wastes DRAM. He's right, and on short chats he undersold it, but every RTX 5090 on Earth is two or three days of Nvidia's racks.

MesmerTools•November 8, 2026•4 min

In October, on Stripe's Cheeky Pint podcast, Sam Altman said this about local inference:

But from a global allocation perspective, actually it's quite terrible to have everybody have a bunch of the world's limited supply of DRAM sitting in their own computer. And, being used... 5% of the time.

He wants "maximum utilization in the cloud" instead. It went around as "a terrible allocation of scarce resources". That's a fair summary. So I ran his numbers on the strongest home case, an RTX 5090.

His math holds. On short chats he even undersold it: a home card makes about 0.3% of what it could in a day. It's still the wrong lever for a shortage, because every 5090 on Earth adds up to two or three days of Nvidia's rack production.

A vast field of unlit LED cells representing one RTX 5090's daily capacity on short chats, with a few tiny amber specks lit, next to the headline Your GPU: 0.3% used

On short chats, local inference wastes more than he said

Writing a token means reading every weight in the model out of memory, while the math units mostly wait. Batching fills that wait: 32 conversations ride one read of the weights, like a bus that's slower with 32 riders, but nowhere near 32 times slower.

I found a published run on one 5090 where 32 short requests made 19 times the tokens of one.

The same flood of weights buys one token or 32, and each user keeps about 60% of solo speed. Measured on an RTX 5090 with about 700 tokens of context per user, slowed 100x.

I took his 5% at face value, but it only counts idle hours. Even while you're chatting, the card runs at a nineteenth of its output. Multiply the two and you get that 0.3%. This grid is the day, to scale.

Find the six lit cells out of 2,280. A card you bought for games would exist anyway, so this overcharges it.

HBM should sink him, and it doesn't

But datacenters burn DRAM too, and that's the obvious way to sink him. Datacenter GPUs use HBM, stacked DRAM that costs about three times the silicon per bit. Here's where the wafers go.

By the end of 2026, HBM takes about 22% of the big three memory makers' wafers and yields about 9% of all bits.

Wafers are scarce whoever's eating them, so the fair test is tokens per wafer. I fit a simulation to those published runs, and flat out, a 5090 beats a GB300, Nvidia's datacenter GPU, two to three times.

Now watch what a normal day does to that lead, with your card on one stream 5% of the day and the GB300 selling 20% of its capacity.

The home card drains twice, by hours and by streams, and the cloud only once. Simulation, short chats, GDDR7 counted as DDR5.

The cloud ends up about 29x ahead. The whole gap is utilization, the factor he named.

But every 5090 on Earth is two or three days of Nvidia's racks

So he wins on efficiency, and that only matters if there's enough home memory to move. Call it a million RTX 5090s, more than Steam's survey implies. They look enormous until the camera pulls back.

The amber patch is all of them, counted in wafers, with GDDR7 treated as DDR5.

He does have a point with "everybody". About 130 million of these cards would be a tenth of a year's DRAM, a real question for a future that hasn't arrived.

So today the shortage won't notice your card. I still wouldn't buy a 5090 to make tokens cheap, because a busy cloud wins that.

What I don't know is how busy either side really is. His 5% has no source, and my 20% is a new provider's figure, the only one I could find.

More from the Blog