LLM Prompt Caching in Production: Costs, Latency, Traps

Transformer models process every input token in sequence, computing attention across the full context window each time. The intermediate computation — specifically the key value attention matrices for each layer — can be stored and reused when the same prefix appears again. That stored computation is the KV cache.

Provider level KV caching has existed since the first commercial LLM APIs launched. It is fully automatic: the provider caches frequently seen prefixes in memory for a short window, and your application benefits without any configuration changes. The catch is that you have no control over what gets cached, how long it stays cached, or what your cache hit rate actually is.

Provider prompt cache APIs give applications explicit control. You mark sections of the prompt as cacheable, the provider computes and stores the KV state for those sections, and subsequent calls that share the same marked prefix skip recomputing those tokens entirely. Anthropic and OpenAI both expose variants of this API with different mechanics and pricing.

Semantic caching sits one layer higher, in your application tier. Instead of reusing KV computation, it stores prior model responses and serves them when a new query is semantically similar enough to a cached query. No model call happens at all on a cache hit. The savings are real, but so is the correctness exposure.

For provider prompt cache APIs, cached input tokens are priced at 10 percent of standard input token pricing (a 90 percent reduction) on Anthropic's API and at 50 percent of standard input pricing on OpenAI's. The right choice depends on your prefix size and cache hit frequency — Anthropic's economics reward long system prompts cached across many calls; OpenAI's automatic model is simpler to implement.

TTFT improvement follows the same logic. If your system prompt is 2,000 tokens and your user query is 200 tokens, and you cache the system prompt, roughly 91 percent of the input computation is skipped on a cache hit. In practice, TTFT reductions sit between 60 and 90 percent depending on the prefix size relative to the query and network conditions.

Semantic cache hit rates depend entirely on your workload. A customer support bot with a narrow FAQ surface can achieve 70 to 80 percent hit rates with a threshold of 0.92. A general purpose assistant with diverse query patterns may see 10 to 20 percent. The cost savings scale linearly with hit rate — a 70 percent hit rate on an application tier semantic cache eliminates 70 percent of model calls entirely.

The cost model for a production system with all three layers is layered: the provider KV cache reduces baseline cost for repetitive system prompts without effort; the prompt cache API compounds those savings for large fixed contexts; the semantic cache eliminates calls outright for repetitive queries. Each layer targets a different component of total inference spend.

Each agent in a chain typically maintains its own context window: the original task, its assigned instructions, tool call history, and observations from prior steps. When agents call LLMs independently, the system prompt for each agent is a candidate for prompt cache API caching. A five agent pipeline where each agent has 4,000 tokens in its system prompt can cache all five system prompts and save meaningful cost on every orchestration run.

The complication is context sharing. When an orchestrator agent passes its full conversation history to a subagent, that history changes with every step, which means the prompt prefix changes and the cache key changes. You need to separate the stable parts of a multi agent prompt (the agent's persona, its tool definitions, its operating constraints) from the dynamic parts (the conversation history, the current task state) and place the cache boundary between them.

For agentic cost optimization more broadly, prompt cache APIs are one of the highest-leverage levers — they require no changes to agent logic, only to prompt structure. See the Agentic AI Cost Optimization: Cut Token Spend at Scale post for the full prioritized lever stack in an agent context.

Provider automatic KV caching requires no effort and delivers baseline savings for any workload with repeated system prompts — enable it by default by structuring your prompts with a stable system prefix. Prompt cache APIs are worth the implementation cost for any workload with a large, stable context (long system prompts, large tool definitions, retrieved documents that remain constant across a session). The ROI is proportional to prefix size and call volume.

Semantic caching is worth building for workloads with demonstrably repetitive query patterns. Measure your actual query distribution first. If more than 30 percent of queries are semantically similar to a prior query in the last 24 hours, the semantic cache will pay for itself. Below that threshold, the embedding and retrieval overhead may not justify the complexity.

The combination of all three layers — provider KV cache, prompt cache API for large stable prefixes, and semantic cache for repetitive queries — can reduce total inference cost by 70 to 90 percent for the right workloads. Measure each layer independently so you know which lever is driving the savings. For a deeper look at the full cost optimization surface including context compression, model routing, and memory pruning, the LLM Inference Cost Optimization: A Production Playbook is the companion reference. For the application level semantic cache layer specifically, LLM Semantic Caching: Cut Latency and Cost at Scale goes deeper on embedding choice, threshold calibration, and hit rate monitoring.

If you are architecting LLM cost and latency infrastructure across a production system and want a second opinion on your caching strategy, that is a standard part of an AI Systems Architecture engagement.