Explainers

KV cache explained: what prompt caching saves, and what it can't

A KV cache is a model's working memory for one response. Prompt caching keeps it between requests: why it's cheap, why it expires, why long chats cost more.

MesmerTools•November 14, 2026•6 min

A KV cache is a model's working memory for the text it has already read. For every token, in every layer, a transformer computes a key and a value, and each later token looks those up to decide what to pay attention to. Earlier tokens never change, so the model computes their keys and values once and keeps them while it writes. That stash is the KV cache, and normally it lasts for one response.

Prompt caching is the provider keeping that memory after the response ends. When your next request starts with exactly the same tokens, the model reads the saved entries instead of recomputing them, and you pay about a tenth of the input price for that part (a twentieth on Claude Sonnet 5.5, Claude Opus 5.5 and GPT-6.1 Sol). It only works on an identical prefix, and it expires within minutes. It makes long conversations cheaper without making them cheap: each request still costs more than the last.

A row of blank token chips with two amber blocks under each; the first ten are grouped inside an outline, with an arrow curving back to the start of the row, beside the headline KV cache, measured

What the KV cache stores, and why the model needs it

Attention is how a token decides which earlier tokens matter. Each token produces a query (what it's looking for), a key (what it offers) and a value (what it passes along). To write the next token, the model compares its query against the keys of everything before it and blends the values of the ones that match. It does that in every layer, dozens of times per token on a large model.

A token only ever looks backward, so its key and value never change once they're computed. Without a cache, the model would recompute them for the whole prompt every time it wrote a word, and a 1,000-token answer would redo the prompt a thousand times. With the cache, each new token computes its own entry and reads the rest.

That splits every request into two phases you can feel. Prefill reads the whole prompt in one parallel pass and fills the cache, which is the pause before the first word appears, and it grows with the length of the prompt. Decode then writes one token at a time, each one reading the cache and adding a single entry.

Watch the cache row: prefill fills it in one pass, each new token reads all of it and adds one entry, and the next request only computes the part that changed. Change the very first token and nothing after it can be reused.

Prompt caching keeps that memory between requests

The model remembers nothing between requests, as the post on how LLMs work walks through. Your next message sends the whole conversation again, and without caching the model reads all of it from scratch. Prompt caching keeps the cache entries on the provider's side, filed under the exact tokens that produced them. A request that starts the same way picks them up and only computes what's new.

It has to be a prefix match. A token's key and value depend on every token before it, so changing one word near the top changes every entry after that word. A timestamp at the start of the system prompt, a tool list that comes out in a different order, or an edited earlier message each turn a cache hit into a full-price request. Appending to the end only costs you the new tokens; editing anything costs you everything after the edit.

It expires quickly because it's big. On Qwen3.8 27B, an open model, each token of context takes 64 KB of cache, so a 128K-token prompt occupies 8 GB of GPU memory while it's in use. That's after Qwen made three of every four layers keep a fixed-size summary instead of a per-token cache; with a cache in every layer it would be 32 GB, more than half the size of the model's own weights. Holding that around isn't free, so lifetimes are short. As of October 2026, Anthropic keeps an entry for 5 minutes after its last use, or an hour if you pay double to write it, OpenAI's current models keep it for 30 minutes, and DeepSeek moves it to disk and clears it within hours to days.

Storing it also costs extra. Anthropic and OpenAI bill the request that writes the cache at 1.25 times the normal input price, so caching a prompt you never send again costs 25% more than not caching it. The first reuse pays that back many times over. OpenAI turns the write on by default, which is the catch in OpenAI API pricing.

One 100,000-token prompt on Claude Sonnet 5.5, at list prices

Sent without caching
$0.20
First send, writing the cache
$0.25
Repeat, reading the cache
$0.01
The write costs a quarter more than plain input; every hit after it costs a twentieth. Most other Claude and OpenAI models charge a tenth on a hit.

How to make the cache hit

Order the prompt from most stable to least: tool definitions and the system prompt first, then long documents, then the conversation, then the new question. Anything that changes per request, like the date, a request ID or retrieved snippets, belongs after the last part you want cached. Switching models mid-conversation starts from zero too, since each model keeps its own cache.

On Claude you mark the end of the reusable part with cache_control or turn on automatic caching for the request, and prompts below a minimum length aren't cached at all (512 tokens on the newest models, more on older ones). OpenAI and DeepSeek cache automatically once a prompt is long enough. Whichever you use, read the usage numbers on the response: Claude reports cache_read_input_tokens, OpenAI cached_tokens, DeepSeek prompt_cache_hit_tokens. If that number stays at zero on a prompt you keep repeating, something near the top is changing.

Long conversations still get more expensive every turn

Agents lean on this harder than anything else, because they resend everything constantly. Every tool call ends one request and starts the next with the whole history attached. In five months of my own Claude Code logs, 96% of all tokens were cache reads, and the same work billed as fresh input would have cost about six and a half times as much. I break that bill down in the Claude Code pricing post.

Caching flattens the climb of a long session without stopping it. Each request still carries the entire conversation, billed at the cache-read price, plus whatever is new at the full write price. As the context grows, every request grows with it.

What one request cost, by how far into the session it came (Opus 5.5, median)

With prompt caching

Requests 2-1045K context
$0.04
Requests 11-50121K context
$0.06
Requests 51-100234K context
$0.08
Requests 101-200387K context
$0.10
Requests 201-400498K context
$0.12

Same tokens, no cache

Requests 2-1045K context
$0.20
Requests 11-50121K context
$0.51
Requests 51-100234K context
$0.96
Requests 101-200387K context
$1.57
Requests 201-400498K context
$2.01
Mind the scales: both sides climb as the conversation grows, but the left one tops out at 12 cents and the right one at $2. From about 15,000 main-conversation requests in my Claude Code logs, priced at API list rates.

In my Opus 5.5 sessions, a typical request early on (the 2nd to 10th) carried about 45,000 tokens of context and cost 4 cents. Between the 201st and 400th the context was around 500,000 tokens and a request cost about 12 cents, three times as much. Without caching, the same climb would have gone from 20 cents to $2.01, ten times as much.

That's the money case for starting a fresh session when the task changes: it goes back to about 4 cents a request instead of 12.

Why the first request after a break costs $2

The discount runs on a clock, and Claude Code runs the longest one. Its main conversation writes to Anthropic's one-hour cache (99% of its writes there; most subagent writes use the five-minute one), and every request refreshes the timer. Keep working and the whole history stays cached, so each request is a cheap read. On Opus 5.5, a request that came within five minutes of the previous one cost 9 cents on average.

Walk away for more than an hour and the entry is gone. The next request has to write the entire conversation into the cache again, at $8 per million tokens on Opus 5.5 against $0.20 to read it, which is 40 times as much per token. The 67 requests in my logs that came after an hour or more away carried about 285,000 tokens of context, close to the 251,000 of the quick ones, and cost $2.06 on average, because 92% of them had to write most of that context again.

Gaps between five minutes and an hour were mostly harmless: the one-hour cache was still warm, and only 3% of those requests had to rewrite. If you call the API directly, the default lifetime is five minutes, so the cliff comes much sooner unless you ask for the one-hour option.

The charge is a one-time cost, because that request leaves the cache warm again and the next ones are back to cents. When you return to a big session, the choice is between paying about $2 to continue it and starting over, where the first request carries only the system prompt and tools (about 30,000 tokens, roughly 25 cents to write). If the next task builds on the old one, resume. If it doesn't, start fresh.

More from the Blog