Skip to content
Vertex Stack
Menu

Two build engagements per year

AI · 7 min read

Prompt caching strategies that cut Claude API bills by 70%

Prompt caching is usually the biggest cost lever on RAG and agent workloads. Here's the math, the right cache_control placement, and the traps that quietly tank cache hit rates.

By Adil Mir · · Updated

Prompt caching is the most under-used cost lever on LLM workloads. Every provider ships it, the discounts are enormous (90% off cached reads on Anthropic and Gemini 2.5+, 50–90% off on OpenAI depending on the model), and yet many production systems leave a large share of the savings on the table because of placement mistakes, bad TTL choices, or cache-busting prompts. This is a walkthrough of how the three major providers implement caching in 2026, the cost math that actually matters at scale, the placement rules that determine whether you get a 90% discount or a 10% one, and the patterns that keep cache hit rates high in production.

The three cache models, compared

Anthropic, OpenAI, and Google took different paths to the same feature. Knowing which model you're paying for changes how you structure your prompts.

ProviderActivationCache read discountWrite surchargeDefault TTL
Anthropic (Claude)Opt-in: cache_control markers on blocks, or one top-level cache_control for automatic placement90% off (0.1× base input) on Claude 4.x; deeper on some newer models1.25× base (5-min) or 2× (1-hr)5 min, 1 hr optional
OpenAI (GPT-5)Automatic, from about 1,024 tokens90% off on GPT-5.x (0.1× base input); less on older modelsNone before GPT-5.6; 1.25× base on GPT-5.6 and later5–10 min idle in memory; up to 24 hr with extended retention on supported models
Google (Gemini 2.5+)Implicit auto + explicit API90% off (0.1× base input)Hourly storage fee on explicitImplicit short; explicit configurable

The upshot: Anthropic gives you the most control, with a read discount at least as deep as the others', but you have to think about cache_control placement. OpenAI is the easiest — it just works above 1K tokens — but you get less control over what is cached and for how long. Gemini sits in the middle with the wrinkle that explicit caching has a storage cost, which matters once you're caching large contexts for long periods.

The cost math that matters

The math is simple but the intuition often isn't. Caching charges a write premium once and then gives you a read discount on every subsequent hit within the TTL. On Anthropic's 5-minute cache, a single read after the write already breaks even (1.25× + 0.1× versus 2× uncached) — one of the cheapest break-even points in any cloud infrastructure. Past that, every hit is pure savings.

A worked example

A customer-support assistant on Claude Sonnet 4.6: 50,000 conversations per day, 10 turns per conversation, a stable 25,000-token system prompt plus knowledge base, and a 200-token user turn. That's 500,000 turns per day, each with 25,200 input tokens and — let's say — 300 output tokens.

  • Without caching: 500,000 × 25,200 input tokens = 12.6B input tokens/day. At $3/MTok, that's ~$37,800/day on input alone.
  • With caching (first turn writes, nine turns read): 50,000 × 25,000 tokens written at $3.75/MTok = $4,687, plus 450,000 × 25,000 tokens read at $0.30/MTok = $3,375. Input cost drops to ~$8,062/day — a 79% reduction on input spend.
  • Output cost is unchanged in both cases, so the end-to-end bill reduction is slightly smaller — about 74% in this example once the ~$2,250/day of output (150M tokens at $15/MTok) is added back. The more output-heavy the workload, the smaller the overall reduction.

Where to put cache_control — the rules that matter

Anthropic's caching matches on the exact prefix up to and including each cache_control marker. The order of blocks matters: tools, then system, then messages. You get up to four breakpoints per request, and longer-TTL breakpoints must appear before shorter-TTL ones. Violate any of these and your hit rate silently collapses.

// Tool-calling agent with two cache breakpoints
// BP1: stable agent instructions — 1-hour TTL (rarely change)
// BP2: end of turn N-1 — 5-minute TTL (conversation state)
const response = await anthropic.messages.create({
  model: "claude-sonnet-4-6",
  max_tokens: 2048,
  tools,
  system: [
    {
      type: "text",
      text: AGENT_INSTRUCTIONS,
      cache_control: { type: "ephemeral", ttl: "1h" },
    },
  ],
  messages: [
    ...priorTurns.slice(0, -1),
    {
      ...priorTurns.at(-1)!,
      // Mark the last prior turn with a 5-min breakpoint
      // so the next request reads everything up to here
      content: attachCacheControl(priorTurns.at(-1)!.content, "5m"),
    },
    { role: "user", content: newUserTurn },
  ],
});
 
// On every response, check usage to confirm cache hit
// response.usage.cache_read_input_tokens > 0 → hit
// response.usage.cache_creation_input_tokens > 0 → write
  • Put the most stable content first — agent instructions, tool definitions, pinned knowledge base. Anything that changes per request goes at the end, after your last cache_control marker.
  • Use a 1-hour TTL for prefixes that change less than hourly (system prompts, knowledge bases). Use the 5-minute default for conversation state. Order breakpoints with the longer TTL first.
  • Never interpolate a user ID, timestamp, or request ID into the cached prefix. One character difference and the cache misses.
  • Verify hits in production. Log cache_read_input_tokens and cache_creation_input_tokens on every response. If your cache hit rate is below 70% on a steady-state workload, something is busting the prefix.

The traps that silently tank cache rates

Most caching bugs fall into a small set of predictable traps. Every one of them shows up the same way in metrics: cache_creation_input_tokens is high, cache_read_input_tokens is low, and the bill doesn't drop. Here's the checklist to run through first.

  1. Dynamic content in the system prompt. A current-date string, a user name, or a request ID interpolated into the system prompt means every request has a different prefix and every request writes a new cache entry.
  2. Non-deterministic tool ordering. If your tool array is built from an object's values, Node.js and browser engines can return different orders across runs. Sort tool definitions by a stable key before sending.
  3. Breakpoint after every message. More breakpoints isn't better — each one is a separate cache entry with its own write cost. Two well-placed breakpoints usually beat four sloppy ones.
  4. 5-minute TTL on high-value stable content. If you're paying to write a 50K-token system prompt and the traffic is bursty, the 1-hour TTL pays off fast. Run the math on your actual traffic pattern, not a steady-state assumption.
  5. Cache content below the minimum. Anthropic's minimum cacheable prefix is model-dependent — 1,024 tokens on Sonnet 4.6, and up to 4,096 on some models — before cache_control is honoured. Short prefixes aren't cached, even if you mark them.

Stacking caching with batching

Batch APIs give you roughly 50% off for latency-tolerant workloads — overnight embeddings, bulk classification, backfills. Anthropic's batch and cache discounts stack. A workload that can use both — say, a nightly re-classification run against a stable prompt — can land at 95% off input token cost on the cached portion versus the naive baseline. That figure comes straight from multiplying the two published discounts (0.5 × 0.1); uncached tokens and cache writes still cost more.

When caching hurts you

Caching is net-negative in a small but real set of situations. If your prompt prefix is genuinely different on every request — heavy personalisation, per-request retrieval where the retrieved chunks change every turn, one-off completions — you're paying the write surcharge on every call and getting no reads in return. Anthropic's 5-minute cache breaks even after a single read, but if a prefix is written and never read again, caching costs you 25% more than not caching.

  • If over 30% of your requests are the first-ever request for a given prefix, review whether caching is helping. Add caching to the stable part (system prompt, tool definitions) and leave the dynamic part uncached.
  • If TTLs are timing out between requests on a low-traffic endpoint, either batch requests within the TTL window or drop to the 1-hour tier.
  • If the prompt is below the minimum size, there's nothing to optimise — caching won't activate at all.

A migration playbook

For an LLM workload that isn't caching yet, the playbook is the same every time. It is usually a day or two of work, and on a workload with a large stable prefix it can remove most of the input token spend on the first pass.

  1. Instrument first. Log cache_read_input_tokens and cache_creation_input_tokens on every response. You can't tune what you can't see.
  2. Identify stable prefixes. System prompts, tool definitions, pinned knowledge — anything that doesn't change per request. Move them to the top of the prompt if they aren't already.
  3. Eliminate accidental dynamism. Strip timestamps, UUIDs, user-specific data from the stable section. If you need to include user context, put it after the last cache_control marker.
  4. Add cache_control markers. Start with one at the end of the system prompt, and — for multi-turn flows — one at the end of the prior message. Resist the urge to add more until the first pass is proven.
  5. Verify. Cache hit rate on steady-state traffic should be north of 80%. If it isn't, something in the prefix is varying and you haven't found it yet.

Key takeaways

  • Caching is usually the biggest cost lever on LLM workloads. On RAG and agent workloads with a large stable prefix, it can remove most of the input spend.
  • Anthropic gives the most control; OpenAI is automatic, with a similar read discount on GPT-5-family models; Gemini sits between, with an hourly storage fee on explicit caches. Pick the provider based on workload, not on caching alone.
  • cache_control placement is rule-driven, not heuristic. Stable content first, longer TTL before shorter, never interpolate dynamic values into the cached prefix.
  • Verify in production with cache_read and cache_creation counters. Cache hit rate below 70% on steady-state traffic means something is silently cache-busting.
  • Stack caching with batch APIs for async workloads — on Anthropic, a cached read through the Batch API costs about 5% of the base input price.