How do LLMs work? One loop explains almost everything they do
How do LLMs work? They predict one token, append it and repeat. Real probabilities from a real model show how that loop explains chat, tools and hallucination.
A large language model does one thing. It reads some text and gives every possible next token, a word or a piece of one, a probability. The software around it picks one of those tokens, adds it to the end of the text, and runs the model again on the slightly longer text. A reply from ChatGPT or Claude is that loop, run a few hundred times.
It sounds too simple to account for what these models do, but it accounts for most of it: why a chatbot states wrong facts with full confidence, how it uses tools, why “thinking” helps and why long chats cost more.

How LLMs work: one token at a time
That run is real. I asked GPT-4o mini, whose API still hands back its probabilities, “Why is the sky blue? Answer in one sentence.” and saved the top candidates at every step. Step through it, or click any token:
Prompt: Why is the sky blue? Answer in one sentence.
Underlined tokens were picked with less than 50% probability. ␣ marks a space that’s part of the token. Click any token to jump there.
Step 6 of 26: what comes next?
Every other token in the vocabulary, about 200,000 of them: 1.7% combined.
Picked “spaceof”. The draw took option #3, at 14.9%. Less likely tokens win sometimes.
Most steps aren’t close. After “The”, the model put essentially 100% on “sky”. The interesting steps are the forks. At step 6 the favorite was “molecules” at 41%, then “shorter” at 36%, and “of” came third at 15%. The pick is a weighted random draw, and a 15% option wins about one time in seven.
That one draw decided the rest of the sentence. After “because of”, the model put 98% on “Ray” and then 100% on “leigh”. A different draw at step 6 would have produced a different sentence, about molecules. Each token becomes input for the next step, so early picks steer everything after them.
What a token is, and why models can’t spell “strawberry”
A token is the unit the model reads and writes. Common English words are usually one token, space included, and rarer words get split into pieces, like “Ray” and “leigh” above. Each model family has its own vocabulary, about 200,000 pieces in OpenAI’s case, and at every step the model scores all of them.
That explains the old trick of asking a model how many r’s are in “strawberry”. To OpenAI’s tokenizer, “strawberry” with its leading space is a single token. The model never sees the letters. It can only answer from whatever it picked up about how that token is spelled, like someone who has heard a word a thousand times and never seen it written down.
Tokens are also what you pay for, and they aren’t a fixed unit. The sky sentence is 26 tokens for OpenAI, 27 for Qwen3 and 55 for Claude Sonnet 5.5. Over a full page of English, Claude’s count came out about 1.4 times OpenAI’s, so two models with the same price per token can differ by 40% on the same text. Compare prices on your own text; our token price calculator does the arithmetic across models.
Where the probabilities come from
Training is the same guessing game. Take trillions of tokens of text, hide the next token, have the model guess, and nudge its billions of internal numbers (the weights) toward the right answer. Predicting the next word of a textbook, a code file or a contract well takes grammar, facts, style and some reasoning, so the model picks all of those up along the way.
The model itself is a transformer: a stack of layers in which each token looks back at every earlier token and pulls in whatever is relevant to it (that’s “attention”). The last layer produces one score per vocabulary entry, and those scores become the probability list you just stepped through. There’s no database of facts in there. Facts are stored the way patterns are, spread across the weights.
A model fresh out of that stage only continues text. Ask it a question and it might reply with three more questions, because that’s how plenty of web pages continue. A second round of training, on example conversations and feedback, shifts the odds so the likeliest continuation of a question is a helpful answer. The loop stays the same; only the probabilities change.
Temperature is how the pick gets made
The model only hands over probabilities. How one token gets picked from them is a setting called temperature. At 0, the app always takes the top token. At 1, it draws from the model’s probabilities as they are. Above 1, it flattens them first, so unlikely tokens get a bigger share. Here’s GPT-4o mini asked for a six-word story about a lighthouse, ten times at each of three temperatures:
Temperature 0
1 different out of 10
- Lonely lighthouse beacon guided lost ships.
- Lonely lighthouse beacon guided lost ships.
- Lonely lighthouse beacon guided lost ships.
- Lonely lighthouse beacon guided lost ships.
- Lonely lighthouse beacon guided lost ships.
- Lonely lighthouse beacon guided lost ships.
- Lonely lighthouse beacon guided lost ships.
- Lonely lighthouse beacon guided lost ships.
- Lonely lighthouse beacon guided lost ships.
- Lonely lighthouse beacon guided lost ships.
Temperature 1
10 different out of 10
- Lonely lighthouse beacon guided ships home.
- Beacon flickered; ship vanished into darkness.
- Beacon flickered; ships found safe harbor.
- Storm raged, but the lighthouse stood.
- Guiding ships home, the lighthouse flickered.
- Storm raged; the lighthouse stood firm.
- Beacon flickered; hope dimmed with darkness.
- Storm raged, light flickered, hope remained.
- Lonely lighthouse stood, guiding ships home.
- Guide in darkness, keeper of hope.
Temperature 2
10 different out of 10
- Its beacon called; waves silently warned.
- Wheeluzioni flashed heartbreak; coordinates whispered hope.T…
- Waves crashed; the light stood sturdy alone.
- Lonely lighthouse shone; ships navigated safely.
- Lost bevGer staying, Wissenschaft yêu Taw天下 留 ele 大发云加勒比 why…
- Waves crashed; beams shone, guiding weary sailors.
- Storm raged; beacon flickered, but unwavering.
- Keeper’s150 aniversarioopen葵oz 발견 novéҫ estado alapján drawn…
- Lonely lighthouse beacon; sailors count merci.
- Engineer quit; solo keeper saved ships.
At 0, the same story ten times. At 1, ten different stories, all fine, which is why asking a chatbot the same thing twice gets you two answers. At 2, seven still work and three dissolve into a jumble of languages. Flattening hands real probability to the long tail of junk tokens, and once one junk token gets drawn, it’s part of the text. From then on the model is predicting what follows garbage, which is more garbage.
Hallucination is the loop doing its job
Nothing in the loop checks whether the model knows something. Every step has to put its probability somewhere, and some token always wins. I asked the same model where six 19th-century scientists who worked in Germany were born, and kept the probabilities for the first token of each answer. Here are three of them, with Marie Curie as a control:
Marie Curie
Sure, and rightSaid Warsaw · born in Warsaw
Friedrich Wöhler
Unsure, and rightSaid Eschersheim · born in Eschersheim
Adolf Hurwitz
Unsure, and wrongSaid Berlin · born in Hildesheim
Ernst Steinitz
Sure, and wrongSaid Berlin · born in Laurahütte
Curie is easy: 99.9% on “Wars”, the start of Warsaw. Friedrich Wöhler got the right town, Eschersheim, from a 33% guess. Adolf Hurwitz is the visible kind of failure: the probability is spread thin, and the top pick, Berlin, is wrong. Ernst Steinitz is the one to worry about. The model put 95% on Berlin, nearly as sure as it was about Curie, and he was born in Laurahütte, in Silesia.
Berlin was its top answer for three of the six, and none of those three was born there. When a specific fact isn’t in the weights, the model falls back on the most typical continuation of “the German mathematician was born in”, and the most typical answer is the capital. A model’s confidence tells you how typical an answer is, which is only loosely related to whether it’s true.
That’s also why the fix that works is putting the facts in front of it. Web search, retrieval and file uploads all paste the source into the text the model is reading, and copying a fact from a few paragraphs up is a far easier prediction than recalling a rare one.
A chat is one long document
A chat model has no memory between messages. Every time you send one, the app rebuilds the whole conversation as a single document, with special tokens marking who said what, and asks the model to continue it. Here’s a real exchange with Qwen3, an open model whose chat format is public, laid out the way the model reads it. I asked whether to bring an umbrella in Kathmandu and gave it a weather tool:
<|im_start|>system # Tools You may call one or more functions to assist with the user query. You are provided with function signatures within <tools></tools> XML tags: <tools> {"type": "function", "function": {"name": "get_weather", "description": "Current weather for a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}} </tools> For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags: <tool_call> {"name": <function-name>, "arguments": <args-json-object>} </tool_call><|im_end|> <|im_start|>user Should I bring an umbrella in Kathmandu today?<|im_end|> <|im_start|>assistant <tool_call> {"name": "get_weather", "arguments": {"city": "Kathmandu"}} </tool_call><|im_end|> <|im_start|>user <tool_response> {"sky": "light rain", "temp_c": 19} </tool_response><|im_end|> <|im_start|>assistant Yes, you should bring an umbrella in Kathmandu today as there is light rain and the temperature is 19°C.<|im_end|>
The model wrote only the amber parts, a token at a time, and ended each one by predicting <|im_end|>, its token for “I’m done”. The app assembled everything else. A chatbot “remembers” what you said earlier only because it’s still in the document. When a conversation outgrows the model’s context window, the app has to drop or summarize something, and whatever leaves the document is gone.
Tool use works the same way. The tool’s description is plain text at the top. The model “calls” the tool by writing the text of a call in the format the instructions asked for. The app spots that text, stops the loop, runs the real function, pastes the result in as another message and starts the loop again. The model never touches the weather service. A coding agent is this cycle on repeat: write a command, read its output, write the next one.
Thinking is the model writing notes to itself
Reasoning models add a section before the answer where the model works through the problem in writing, often for thousands of tokens. It helps for a mechanical reason. The model spends roughly the same computation on every token, so a hard question answered in a few tokens gets only a few tokens’ worth of thought. Writing out the steps buys more passes through the model, and every step it writes down stays in the document for later steps to read, like scratch paper. Those thinking tokens come out of the same loop, so they’re billed as output, even when the app hides them from you.
Why the bill looks the way it does
The model reads all of the input in one parallel pass, but output has to come out one token at a time, each needing its own trip through the model. That’s a big part of why output tokens usually cost two to five times as much as input.
And because the model has no memory, every request sends the whole document again. In the umbrella exchange the model wrote 50 tokens, but the two requests billed 364 input tokens, because the document was read twice and the tool instructions alone were 138 tokens of each read. A long chat works the same way: message 40 pays to re-read messages 1 through 39. Prompt caching makes those re-reads cheaper, as the KV cache post measures, but a long conversation still costs more with every message, and for a coding agent it’s most of the bill, as I show in Claude Code pricing.
So when a model does something odd, skip the question of how smart it is and ask two others: what was in the document it was reading, and what would the most typical next token have been?
More from the Blog
How Diffusion Models Work: A Game Boy Shows They Decide Early
I changed a Game Boy diffusion model's subject mid-drawing. By step 6 of 16 the new request only ties the old one, and the canvas is still 83% noise.
This Seedance AI Film Cost $28,775: Half the Runtime, Same Bill
This Seedance AI film cost $28,775, about what a film nearly twice as long cost. The team got better at retries, but the episode bill didn't shrink.