What is prompt caching?
Prompt caching lets a model provider reuse the work of processing the beginning of a prompt when later requests start with exactly the same content. Repeated system prompts, tool definitions, documents and conversation history are then billed at a steep discount and processed faster. It works on prefixes, so the order of your prompt matters.
Most tokens in a production LLM request are ones you already sent a minute ago. The system prompt, the tool list and the conversation so far repeat on every call. Prompt caching stops you paying full price to process them again.
How does prompt caching work?
When a model reads a prompt, it computes internal attention values (the key-value, or KV, cache) for every token. Normally that work is thrown away after the request. With prompt caching, the provider stores the computed state for the start of your prompt. If a later request begins with exactly the same tokens, the model loads the stored state instead of recomputing it and only processes the new part at the end.
The match is on an exact prefix. Change one character near the start, and everything after it misses the cache.
| Request | Content | What is cached |
|---|---|---|
| 1 | System prompt + tools + document + question A | Nothing yet; the prefix is written to the cache |
| 2, a minute later | Same system prompt + tools + document + question B | Everything up to the question is read from cache |
| 3 | System prompt with today's date added at the top + ... | Cache miss from the first changed token onwards |
How much does prompt caching save?
Savings are large, though exact terms vary by provider and model and change over time. At the time of writing, Anthropic charges cache reads at about a tenth of the normal input price, with cache writes costing somewhat more than normal input, and a default cache lifetime of five minutes that refreshes each time it is used. OpenAI applies caching automatically on supported models for prompts above a minimum length, with discounts of up to 90% on cached input on recent models. Google's Gemini API offers both automatic and explicit context caching. Providers also report substantially lower latency on long cached prompts, because less input has to be processed before the first output token.
Check the current pricing page before estimating costs, and read the usage fields in each API response, which report cached and uncached token counts separately.
How do you structure a prompt for caching?
Put content that never changes first and content that changes on every request last.
- 1Tool definitionsStable across requests, often thousands of tokens for agents.
- 2System promptKeep it identical: no timestamps, request IDs or user names at the top.
- 3Long reference materialDocuments, codebase summaries or few-shot examples reused across questions.
- 4Conversation historyGrows turn by turn; earlier turns stay a valid prefix.
- 5The new user messageAlways last, since it changes every time.
With Anthropic's API you mark where the cacheable prefix ends with a cache_control breakpoint; OpenAI caches matching prefixes automatically. Either way, the order above is what makes cache hits possible.
When is prompt caching most useful?
- Do: Agents, which re-send a long, growing history and a large tool list on every step
- Do: Chat over a long document, where many questions share the same document prefix
- Do: Applications with a long system prompt and many short user requests
- Do: Batch jobs that run the same instructions over many inputs in quick succession
- Avoid: One-off requests with no repeated prefix
- Avoid: Prompts shorter than the provider's minimum cacheable length
- Avoid: Prompts that put a changing value, such as the current time, before the stable content
Is prompt caching the same as caching responses?
No. Prompt caching reuses the processing of the input; the model still generates a fresh answer every time, and the answer is billed normally. Response caching, which you build yourself, stores whole answers and returns them for repeated identical or near-identical questions without calling the model at all. The two can be combined. See LLM cost and latency for other savings.
Frequently asked questions
Does prompt caching change the model's answers?
No. The cached state is exactly what the model would have computed anyway, so outputs are the same as without caching. Only cost and speed change.
How long does a prompt cache last?
It depends on the provider and settings. Anthropic's default is five minutes, refreshed on each use, with a longer paid option. OpenAI's lifetime varies by model and is typically minutes, with extended options. Check current documentation.
Why am I not getting cache hits?
The most common causes are content that changes near the start of the prompt, such as a timestamp in the system prompt, prompts below the minimum cacheable length, or requests spaced further apart than the cache lifetime.
Is prompt caching private?
Providers scope caches to your organisation, so other customers cannot read your cached prompts. Check your provider's documentation for details on data retention.
Last checked for accuracy on . Written by the solidcoder team.