Section 01 · Definition
What is LLM semantic caching?
Traditional exact match caching only helps when a user sends the identical string twice. Semantic caching handles the far more common case where two prompts mean the same thing.
Quick answer
What is semantic caching for LLMs? Semantic caching for LLMs is a layer that intercepts a prompt, converts it to an embedding, measures its cosine similarity against previously stored prompt embeddings, and returns the cached response when the similarity exceeds a configured threshold — bypassing the language model entirely.
The key word is semantic. Traditional exact match caching only helps when a user sends the identical string twice. Semantic caching recognises that “What is RAG?” and “Can you explain retrieval augmented generation?” are asking the same thing and should return the same answer. The cache stores the embedding vector of the original prompt alongside the response. Each new prompt is embedded at query time, the nearest stored vector is retrieved, and the distance is checked against the threshold. Above the threshold: cache hit. Below: cache miss, LLM called, new response stored.
The mechanism looks straightforward. The engineering is in the details — specifically the threshold, the invalidation strategy, and the decision about which responses are safe to cache at all.
Section 02 · Mechanism
How it works: embeddings, cosine similarity, and the lookup path
Every semantic cache follows the same four steps. Understanding each one is where the implementation choices live.
Embed the incoming prompt
The query is passed to an embedding model — text-embedding-3-small, Cohere Embed, or a locally hosted model. This produces a dense vector of several hundred to a few thousand dimensions encoding the prompt's semantic meaning.
Search the vector index
The embedding is compared against all stored prompt embeddings using approximate nearest-neighbour search. Redis Vector, Qdrant, Pinecone, and Weaviate all support this. The search returns the top-K nearest stored prompts along with their cosine similarity scores.
Apply the threshold
If the highest cosine similarity score exceeds the threshold — commonly set between 0.90 and 0.98 — the cache returns the stored response. The threshold is the central design decision. Too permissive risks wrong answers; too strict defeats the purpose.
Serve or populate
On a hit, the stored response is returned with latency in the 10 to 50 millisecond range. On a miss, the LLM is called normally and the new prompt embedding plus response are written to the cache store.
The embedding step adds a small amount of latency on both hit and miss paths — typically 20 to 80 milliseconds depending on the embedding model and whether it is hosted locally or called via API. For cache misses this is negligible. For cache hits with a fast embedding endpoint, the round trip is still five to ten times faster than a median frontier model call.
Section 03 · Production Data
What the numbers say
The published benchmarks cluster around a few consistent findings. The big caveat: workload type determines everything.
| Workload type | Typical hit rate | Avg latency gain | Token cost reduction |
|---|---|---|---|
| Customer support Q&A | 61–68% | 5–10× avg | ~60–65% |
| Document question answering | 55–65% | 4–8× avg | ~55–62% |
| Structured data extraction | 60–70% | 5–12× avg | ~60–68% |
| Conversational chat | 30–45% | 2–4× avg | ~28–43% |
| Creative generation | 15–30% | 1.2–2× avg | ~13–28% |
| Code completion | 25–40% | 2–5× avg | ~23–38% |
Latency reduction on a cache hit is dramatic. A hit returning in 20 milliseconds versus a model call taking 1,200 milliseconds represents a 60x speedup for that request. Across a mixed hit and miss traffic stream at 65 percent hit rate, average response time drops by roughly 40 to 50 percent.
Cost reduction follows the hit rate almost linearly. At a 65 percent hit rate, you are paying the LLM for 35 percent of requests instead of 100 percent. If the embedding call costs 1 to 2 percent of what the LLM call would have cost, the net saving on token spend is approximately 63 to 65 percent. One team running GPT-4o at scale reported a 73 percent reduction in model API spend after adding a semantic cache with a 0.92 threshold across their support ticket classification pipeline.
The caveat: these numbers come from high repetition workloads. If your workload is inherently diverse, semantic caching reduces costs by 15 to 25 percent at best, and the engineering overhead may not justify the gain.
Section 04 · Threshold Calibration
Threshold tuning and false-hit risk
The threshold is the single most operationally important parameter in a semantic cache, and it is the part most vendor tutorials skip.
A false hit happens when two prompts are semantically similar enough to exceed the threshold but actually require different answers. The classic example is question answering over a changing knowledge base: “What is the return policy?” asked in January and the same question asked in March will get identical embeddings but the policy may have changed. Another example is user personalisation: “Summarise my account activity” asked by two different users will embed identically but must return different data.
False hits are more dangerous than cache misses. A miss costs you a model call. A hit that serves the wrong answer to the user can cause real harm — wrong information delivered with the confidence of a cached, previously approved response.
Build an offline evaluation before you deploy
Collect a sample of your production prompts. Label pairs as should-share or should-not-share a cache entry. Run the embedding model on the sample, plot the cosine similarity distribution for both classes, and set the threshold at the decision boundary where the two distributions overlap least.
Typical landing zone is 0.90 to 0.96
For most NLP workloads this is where the calibrated threshold falls. For domains where precision matters more than recall — medical, legal, financial — push the threshold above 0.95. The gain in false-hit prevention outweighs the lower hit rate.
Re-evaluate monthly
Embedding distributions drift as user vocabulary and query patterns change. A threshold correct at launch may generate false hits six months later as the product evolves. Schedule a monthly offline evaluation as part of your LLM observability practice.
Section 05 · Invalidation
Cache invalidation and staleness
Unlike a traditional key value cache where an exact key change invalidates a record, a semantic cache has no natural eviction trigger. This is the blind spot most teams hit in month three.
If the underlying knowledge changes — a product price is updated, a policy is revised, a fact is corrected — there is no automatic mechanism to evict the cached responses that answered questions about the old state. The three standard approaches each cover different parts of the staleness risk:
Time to live (TTL)
Set a maximum age on cached entries. Entries expire after the configured period regardless of whether the underlying data changed. Crude but effective for slowly-changing domains. A customer support cache might use a TTL of 24 hours. A real-time financial data assistant should not use TTL based caching at all.
Topic namespace partitioning
Group cached entries by domain — product catalogue, HR policies, legal terms. When a document in a namespace is updated, invalidate all cache entries in that namespace. More precise than global TTL but requires maintenance discipline as content changes.
Source triggered eviction
Wire the cache to the content management system or knowledge base. When a document version changes, issue an eviction command for cached entries whose source document matches. Most accurate but most complex. If your RAG pipeline already tracks source document IDs against retrieved chunks, the plumbing is mostly there.
The hybrid approach that works for most teams: TTL of 24 to 72 hours as a baseline, namespace eviction for high stakes domains (pricing, compliance), and manual flush tooling for emergency corrections. This covers 95 percent of staleness risk at 20 percent of the engineering cost of full source triggered eviction.
Section 06 · Cache Tiers
Verified versus unverified cache tiers
Production teams running semantic caches at scale split them into two tiers. The two tier model is how you keep the cache surface clean as it grows.
| Tier | What goes here | Threshold | Quality gate |
|---|---|---|---|
| Unverified | All LLM responses on first cache miss | Strict (0.96+) | None yet — generated but not reviewed |
| Verified | Responses that passed a quality review | Standard (0.90–0.94) | Human review, LLM judge, or A/B signal |
Queries hit the verified tier first. Only on a miss there do they fall through to the unverified tier, and the unverified tier's stricter threshold is an additional safeguard. The operational benefit is a graceful ramp: day one of deployment, the verified tier is empty and everything falls through to the LLM. Over days and weeks, the verified tier fills with high confidence, high repetition answers. The cache hit rate on the verified tier grows, and the average response quality is always the best answers you have reviewed.
For teams running agentic AI systems where the same tool calls and subqueries repeat across sessions, the two tier model also applies at the subquery level — caching the outputs of deterministic tool calls (database lookups, structured data extractions) separately from the outputs of generative steps (summaries, recommendations) because their invalidation logic differs.
You can read more about the broader LLM observability practices that give you the data to run this kind of tiered quality gate effectively.
Section 07 · When to Skip
When semantic caching is not the answer
Semantic caching compounds returns on high repetition, deterministic workloads. Three situations where it hurts more than it helps.
Creative and generative workloads
If users expect varied, nondeterministic responses to the same prompt — creative writing assistants, brainstorming tools, code exploration — serving a cached response breaks the product contract. The whole value is that the model gives you something new each time.
Personalised responses
Any prompt that implicitly or explicitly depends on user state cannot be safely cached without per user partitioning. Partitioned user caches narrow the hit rate to near zero and add storage complexity that often exceeds the savings.
Low repetition or highly diverse query distributions
A simple experiment: embed your last 10,000 prompts and cluster them. If fewer than 15 percent of prompts fall within a 0.92 similarity radius of another prompt in the batch, your workload is too diverse. Consider model routing or context compaction first.
You can pair semantic caching with model routing for the best of both approaches: the router sends high repetition, factual queries to the semantic cache first, and only routes diverse or creative queries directly to the model tier. The LLM model routing guide covers the routing layer in detail.
Section 08 · Summary
Getting semantic caching right
The case for semantic caching is clear. The risk is equally clear. The teams who get it right treat the threshold as a product decision, not a technical default.
The engineering case for semantic caching is strong: 60 to 70 percent hit rates on the right workload translate directly into roughly 60 percent reduction in model API spend and a meaningful latency improvement for the average user.
The engineering risk is equally clear: a poorly tuned threshold serves wrong answers with false confidence, and an unmaintained cache returns stale facts weeks after the underlying knowledge changed.
The teams who get it right build offline evaluations, start with a conservative threshold, measure false hit rates in production, and tune from data. They wire the cache to the source of truth for invalidation rather than relying on TTL alone. And they use the verified/unverified model to ensure the cache surface always reflects their best answers, not their most recent ones.
If you are building or auditing an AI system and need to evaluate whether semantic caching fits your architecture — alongside LLM cost optimisation levers like model routing and context compaction — the AI Systems Architecture service covers the full production stack.