Prompt caching: cut cost and latency on repeated context
Prompt caching lets the provider reuse the work it already did on the beginning of your prompt, when that beginning is exactly the same as a recent request. Cached input tokens are billed at a large discount and processed faster, so long system prompts, tool definitions, documents and conversation history become much cheaper to resend. It only works on an identical prefix, so put stable content first and changing content last, and check your provider's rules, because some cache automatically and some need you to mark what to cache.
What problem it solves
Many applications send the same long text again and again. A support bot sends the same 3000 token system prompt and tool definitions on every message. A document assistant sends the same 50 page handbook with every question. A chat sends the whole conversation history on every turn.
Without caching, the model processes all of those tokens from scratch every time, and you pay full price for them every time.
How it works
When a model reads your prompt, it computes intermediate results for every token, stored in the KV cache, before it can generate anything. That step, called prefill, is a big part of the latency for long prompts.
If the next request starts with exactly the same tokens, the provider can keep those intermediate results for a while and reuse them instead of computing them again. Only the new part at the end has to be processed.
Two consequences follow from how this works:
- It only applies to a prefix. The cached part must be at the start, and it must be byte-for-byte identical. One changed character early in the prompt means everything after it is processed again.
- It expires. Caches are kept for a few minutes up to longer periods, depending on the provider and settings, so it helps most when similar requests come often.
What it saves
Providers price cached input tokens at a large discount compared with normal input tokens, and cached prefixes also reduce the time to the first token. The exact discount, the minimum prompt length that can be cached, how long the cache lives and whether writing to the cache costs extra all differ between providers and change over time, so check the current documentation.
The savings depend on how much of your prompt is stable and how often it repeats. A chat with a long system prompt and many turns can save a large share of its input cost. A batch of unrelated short prompts saves almost nothing.
Automatic and explicit caching
Providers handle this in two ways.
Some cache automatically: if your prompt is long enough and starts with a prefix seen recently, the discount is applied, and the usage section of the response tells you how many tokens were cached.
Some need you to mark cache points: you add a marker at the end of the stable part of the prompt, like the system prompt or the document, and the provider caches everything up to that point, sometimes with a choice of how long to keep it.
In both cases, look at the usage data in the API response to confirm caching is happening, and log cached tokens separately in your token tracker.
How to structure prompts for caching
Order the content from most stable to least stable:
- System prompt and fixed instructions.
- Tool definitions.
- Fixed examples.
- Large shared documents.
- Conversation history (grows at the end, so earlier turns stay cacheable).
- Retrieved chunks for this question.
- The user's new message.
And avoid the things that silently break the cache:
- Timestamps, request IDs or user names at the top of the system prompt. Put dynamic values near the end instead.
- Tool definitions in a different order on each request.
- Randomly shuffled examples.
- Small edits to the system prompt between requests.
- JSON serialised with a different key order.
Prompt caching is not response caching
Prompt caching reuses the processing of the input, and the model still generates a new answer every time. Response caching stores a complete answer and returns it for the same question without calling the model at all.
Response caching saves more per hit, but only applies when the exact question repeats, or with semantic caching, when a similar question repeats, which carries the risk of returning an answer that doesn't quite fit. Both are useful, for different situations, and they work well together.
When it matters most
- Chat applications with long system prompts and multi-turn conversations.
- Agents, which resend tool definitions and growing history on every step.
- Question answering over one shared document or codebase.
- Coding assistants, which resend large amounts of code context.
From the floor
In the customer support agent, the router checks whether a similar query was already answered and serves it from the cache instead of calling the LLM again, because inference is the most expensive part of the process and I want to limit the queries the LLM is generating. That is response caching, a different layer from the prompt caching on this page, and the two work well together: the response cache skips the call entirely, and prompt caching makes the calls you can't skip cheaper.