Skip to content
Novandri.

Prompt Caching: Reducing Cost on Repeated Context

Many requests share the same long prefix. Read how caching bills that prefix, a worked saving example, and why cache misses happen.

Rachmadwipa Novandri5 min read

Most production LLM applications send the same material at the start of every request — system instructions, tool definitions, a knowledge base, or a long document. Without caching, that prefix is reprocessed and re-billed each time. Prompt caching lets a provider reuse the already-computed representation of a stable prefix.

Direct answer: Prompt caching stores the computed representation of a repeated prompt prefix so subsequent requests with the same prefix can skip recomputation, typically at a lower billed rate. OpenAI and Anthropic both document the mechanism and its conditions. The boundary: caching only helps when the prefix is genuinely stable — a changed token early in the prefix invalidates everything after it.

A four-step flow: keep the prefix stable, vary only the suffix, read cache metrics, then watch the misses.
Conceptual: a stable prefix and a small variable suffix. Reordering a few early tokens invalidates the prefix.

How caching works

A transformer processes a prompt as a sequence, building internal representations layer by layer. If two requests share the first N tokens, the computation for those tokens is identical. A cache stores that intermediate state so the second request can start from token N+1.

Three conditions matter:

The prefix must be identical from the first token. A change at token 5 invalidates the cached state for everything after it. This is why stable material — instructions, tool schemas, a fixed document — goes at the front, and variable material — the user's question — goes at the end.

The cache has a lifetime. Providers expire cached prefixes after a period of inactivity. A request that arrives after expiry pays full price, as the documentation explains.

Breakpoints may be explicit. Some APIs require you to mark where a cache block ends. Placing them well is part of the design.

A worked saving example

The following uses published list prices with a hypothetical workload.

OpenAI lists GPT-6 Astra at $10 per million input tokens; cached input is billed at a lower rate. Suppose an application sends a 100,000-token knowledge base with every request and receives a 500-token answer.

Without caching: 10,000 requests × 100,000 tokens × $10/M = $10,000 for input alone.

With caching at a quarter of the input rate: the repeated prefix costs 0.25 × $10/M = $2.50/M. Input cost becomes 10,000 × 100,000 × $2.50/M = $2,500, plus a small number of cache-write charges on the first request and after expiry.

Three lessons follow:

  1. The gain scales with prefix length and request volume. A short prefix with few requests barely matters.
  2. Cache stability is the whole game. If the knowledge base changes every hour, most requests miss and the saving disappears.
  3. Variable content belongs at the end. Putting the user's question before the document would invalidate the cache on every request.

Reading the cache metrics

Providers report cache behaviour, and a few fields matter more than the headline hit rate:

Cache-read tokens. The number of input tokens served from cache. This is where the saving appears on the invoice.

Cache-write tokens. The cost of storing a new prefix. A prefix that is written once and read a thousand times is ideal; one that expires before the next request is a loss.

Time-to-live and eviction. A cache may expire after inactivity or be evicted under pressure. A high write-to-read ratio usually means the prefix is changing too often or traffic is too sparse.

Prefix stability. Track whether the first few hundred tokens of requests are identical. A single dynamic field at the front — a timestamp, a session ID, a rotating example — can invalidate an otherwise perfect cache.

These fields turn caching from a feature you enable into a system you operate. A team that watches only total input cost may miss that a recent prompt change quietly halved the hit rate.

Where caching pays

Long document question-answering with a fixed corpus and many questions.

Agent systems that resend the same tool definitions and instructions at every step.

High-volume support applications with a stable policy document and varied user questions.

In each case the pattern is the same: a large stable prefix and a small variable suffix.

What caching cannot do

It does not reduce output cost. Generated tokens are still billed at the normal rate, as the cost estimator shows.

It does not survive arbitrary edits.

Versioning the knowledge base deliberately is better than letting it drift.

It is not a guarantee. A cache miss due to expiry or routing means a full-price request. Measure the hit rate, which the observability article explains how to instrument.

The counterargument — that cached input is cheap enough to ignore — is true at low volume and false at scale.

A 100,000-token prefix sent 10,000 times is a billion tokens of input, and the difference between full and cached rates is material.

A decision test

  1. Identify the stable prefix in your requests.
  2. Place variable content at the end of the prompt.
  3. Mark cache breakpoints where the API requires them.
  4. Measure the cache hit rate and the effective cost per request.
  5. Version the prefix deliberately so changes are controlled.
  6. Account for expiry in the cost model.
  7. Re-check rates when the provider updates pricing.

For the broader cost model, see LLM Cost Estimator; for long prefixes, see Long Context Windows.

References & Further Reading