Semantic Caching for AI Products: Cut Inference Costs Without Sacrificing Quality
TL;DR
Semantic caching is an application-layer technique that uses vector embeddings to detect when a new query is semantically close to a previously answered query, and returns the cached response instead of calling the LLM. Done well, it cuts inference costs by 20 to 40% on workloads with repeated query patterns. Done poorly, it silently returns stale or incorrect answers to users who asked a subtly different question. This guide covers how it works, when it makes sense, the three implementation options, and the three failure modes that will hurt you if you miss them.
The AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.
What Semantic Caching Is (and Is Not)
There are three distinct caching techniques in AI systems, and they operate at very different layers. Getting them confused leads to applying the wrong technique to the wrong problem.
KV cache (inference-layer)
Within one inference callStores the key-value attention matrices computed during the forward pass. Speeds up token generation within a single inference call. Built into the transformer runtime. You do not configure this; the GPU does it automatically.
Prompt caching (API-layer)
Across calls sharing the same prefixCaches the computation for a repeated prompt prefix across different inference calls. You pay a lower per-token rate when the prefix was already processed. Anthropic, OpenAI, and Google all support this. Reduces cost when many calls share a long system prompt.
Semantic caching (application-layer)
Across calls with similar meaning, regardless of exact wordingStores complete input-output pairs. When a new query arrives, its embedding is compared against the stored embeddings. If the similarity score exceeds a threshold, the stored output is returned without calling the LLM at all. Operates entirely outside the LLM provider.
Semantic caching is the only one of these that can return a cached answer to a query the user phrased differently from any prior query. "What is the return policy?" and "How do I return a purchase?" are different strings. Their embeddings are close. A semantic cache with a reasonable similarity threshold treats them as equivalent and returns the same answer.
How It Works Under the Hood
The implementation has four steps, each with a design decision that affects both cost and quality.
Embed the incoming query
The user's query is converted to a vector using an embedding model. You want a small, fast model here, not the largest available one. text-embedding-3-small from OpenAI and Voyage-3-lite from Voyage AI are common choices at a cost of roughly 0.02 cents per 1,000 tokens. The embedding model's quality determines how well 'semantically similar' maps to 'should get the same answer'.
Perform nearest-neighbor lookup
The embedded query is searched against a vector database storing past query embeddings. You retrieve the top-k most similar past queries and their similarity scores. This lookup needs to be fast: if the cache lookup takes 200ms and the LLM call takes 400ms, you have cut latency, not doubled it.
Apply the similarity threshold
If the highest similarity score exceeds your threshold, return the cached response. The threshold is your single most important design decision. At 0.99+ you rarely get cache hits. At 0.85 you get many hits but some will be wrong. 0.92 to 0.95 is a reasonable starting range for most customer-facing applications, but this needs empirical calibration on your specific query distribution.
Store new input-output pairs
When the cache misses, the call goes to the LLM. After the response is generated, the input embedding and the response are stored for future cache hits. You need a TTL policy: how long before a cached answer expires. For product FAQ content, 24 to 48 hours. For content that references real-time data, do not cache at all.
When Semantic Caching Is Worth It
Semantic caching has a setup cost, an ongoing maintenance cost for cache management, and a quality risk. It is not worth those costs unless the workload has the right characteristics.
Strong fit
- +Customer support bots where many users ask variations of the same 50 questions
- +Internal knowledge base Q&A where the underlying document corpus changes infrequently
- +Product FAQ and help systems with a bounded domain of questions
- +AI search on a closed corpus where the correct answers do not change day to day
- +Code explanation tools where the same snippets are frequently submitted
Poor fit
- -Conversational assistants where each response depends on prior context in the session
- -Anything referencing real-time or frequently changing data (stock prices, live events, news)
- -Personalized recommendations where the right answer differs by user
- -Creative tasks where users expect a novel output each time
- -Agentic workflows where the LLM is reasoning over a dynamic environment
A rough threshold for viability: if fewer than 15% of your queries are semantically close to any past query at a threshold of 0.92, the cache hit rate is too low to justify the added infrastructure. Run a backward-looking simulation on a week of logs before you build.
Go Deeper on AI Infrastructure Strategy
The AI PM Masterclass covers cost architecture, inference optimization, and the technical decisions that separate AI PMs who understand their stack from those who do not.
Implementation Options: Hosted, Open Source, and DIY
You have three implementation paths, and the right one depends on your team's infrastructure comfort, your existing vendor stack, and how much you want to invest in tuning.
Hosted: Momento Semantic Cache or Upstash Vector
Effort: LowCost model
Pay per GB stored + per query
Best for
Teams that want to ship quickly without managing vector infrastructure
Tradeoffs
Less control over similarity threshold tuning. Vendor lock-in risk. Both have generous free tiers, so this is often the right starting point.
Open source: GPTCache
Effort: MediumCost model
Your own compute + embedding API costs
Best for
Teams with existing ML infrastructure who want maximum customization
Tradeoffs
GPTCache is actively maintained and supports multiple embedding models and vector stores. Requires someone on your team to own the operational overhead. Configuration complexity is high if you need fine-grained TTL policies per query type.
DIY on Redis with vector module
Effort: HighCost model
Redis Enterprise pricing or self-hosted compute
Best for
Teams that already run Redis as their primary cache layer and want to add semantic caching without a new service
Tradeoffs
The tightest integration with existing cache infrastructure, but requires you to implement the embedding step, the similarity search query, and the TTL logic yourself. Not recommended as a first pass.
The Tradeoffs That Will Hurt You If You Miss Them
Semantic caching has three failure modes that are unique to the technique. They are not edge cases. Teams that deploy without accounting for them regularly get burned by one of them within the first month.
Quality degradation from too-low a threshold
HighAt a threshold of 0.87, a query like 'What is your cancellation policy?' may match a cached response to 'How do I cancel my account?' The user was asking about a free trial cancellation; the cached answer was about account deletion. The response is technically about cancellation but wrong for the user's intent.
Mitigation
Start at 0.95 and lower gradually while monitoring a human-rated sample of cache hits. Never lower the threshold based on cost savings alone.
Stale cache serving outdated information
Medium to HighYour product pricing changes. Your return policy changes. Your documentation changes. A cached answer from three days ago is now wrong. Users who asked the question today get the old answer with no indication it might be out of date.
Mitigation
Implement TTL policies per content type. Product pricing: 4 to 8 hours. Policy content: 24 hours. Static reference content: 7 days. Build a cache invalidation mechanism tied to your content update events.
Privacy leakage across users
CriticalUser A queries 'What is my account balance?' and the response includes their specific balance. User B asks the same question. If the cache does not properly scope responses to the requesting user, User B sees User A's account balance. This has happened in production deployments. It is not theoretical.
Mitigation
Never cache responses that include user-specific data. Apply a user context check before every cache write: if the response contains any value from the user's session data, do not cache it. This is mandatory, not optional.
What to Instrument Before You Ship
Semantic caching is invisible to the user. That means failures are also invisible until a user files a complaint or you check the metrics. Ship it with this instrumentation in place from day one.
Cache hit rate by query type
A global hit rate hides the signal. You want to know which query categories are being cached and whether the cached answers in those categories are accurate.
Similarity score distribution
Track the distribution of similarity scores on cache hits, not just the threshold. If most hits are clustering at 0.91 to 0.93 with a threshold of 0.90, you are near the quality cliff and should increase the threshold.
User feedback rate on cached vs. live responses
If users are 3x more likely to give a thumbs-down rating on a cached response than a live one, your threshold is too low regardless of the cost savings.
Cache age at hit
What percentage of your cache hits are serving responses older than 24 hours? Older than 72 hours? This tells you whether your TTL policy is appropriate for your content update cadence.
Set an alert at the engineering level: if cache hit rate on a specific query category drops below 50% of its 7-day average, fire an alert. This usually signals a content change that invalidated a large cluster of cached answers, which means users are getting stale responses.
Master AI Product Architecture Decisions
The AI PM Masterclass covers inference cost management, caching strategies, and the technical depth that lets you make architecture calls with confidence. The November cohort starts Dec 1.
Related Articles
Before you go: get the AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.