Sam Altman on Local Inference: Right Math, Wrong Lever
Sam Altman says local inference wastes DRAM. He's right, and on short chats he undersold it, but every RTX 5090 on Earth is two or three days of Nvidia's racks.
In October, on Stripe's Cheeky Pint podcast, Sam Altman said this about local inference:
But from a global allocation perspective, actually it's quite terrible to have everybody have a bunch of the world's limited supply of DRAM sitting in their own computer. And, being used... 5% of the time.
He wants "maximum utilization in the cloud" instead. It went around as "a terrible allocation of scarce resources". That's a fair summary. So I ran his numbers on the strongest home case, an RTX 5090.
His math holds. On short chats he even undersold it: a home card makes about 0.3% of what it could in a day. It's still the wrong lever for a shortage, because every 5090 on Earth adds up to two or three days of Nvidia's rack production.

On short chats, local inference wastes more than he said
Writing a token means reading every weight in the model out of memory, while the math units mostly wait. Batching fills that wait: 32 conversations ride one read of the weights, like a bus that's slower with 32 riders, but nowhere near 32 times slower.
I found a published run on one 5090 where 32 short requests made 19 times the tokens of one.
I took his 5% at face value, but it only counts idle hours. Even while you're chatting, the card runs at a nineteenth of its output. Multiply the two and you get that 0.3%. This grid is the day, to scale.
HBM should sink him, and it doesn't
But datacenters burn DRAM too, and that's the obvious way to sink him. Datacenter GPUs use HBM, stacked DRAM that costs about three times the silicon per bit. Here's where the wafers go.
Wafers are scarce whoever's eating them, so the fair test is tokens per wafer. I fit a simulation to those published runs, and flat out, a 5090 beats a GB300, Nvidia's datacenter GPU, two to three times.
Now watch what a normal day does to that lead, with your card on one stream 5% of the day and the GB300 selling 20% of its capacity.
The cloud ends up about 29x ahead. The whole gap is utilization, the factor he named.
But every 5090 on Earth is two or three days of Nvidia's racks
So he wins on efficiency, and that only matters if there's enough home memory to move. Call it a million RTX 5090s, more than Steam's survey implies. They look enormous until the camera pulls back.
He does have a point with "everybody". About 130 million of these cards would be a tenth of a year's DRAM, a real question for a future that hasn't arrived.
So today the shortage won't notice your card. I still wouldn't buy a 5090 to make tokens cheap, because a busy cloud wins that.
What I don't know is how busy either side really is. His 5% has no source, and my 20% is a new provider's figure, the only one I could find.
More from the Blog
NVIDIA MotionBricks: The 15,000 fps Isn't the News
MotionBricks' 15,000 fps is mostly a counting choice. I ran it on a laptop. The news is fast, smooth and accurate at once, enough to replace game transitions.
ChatGPT Images 2.6 GIFs Act Like Sprite Sheets: They Keep the Look, Lose Count
ChatGPT Images 2.6 GIFs behave like one sprite sheet cut into frames, so they keep the look and lose count. In 18 juggler runs, none kept three balls.