Semantic Caching for LLMs: How to Cut API Bills by 60% Without Hurting Quality
Learn how semantic caching cuts LLM API costs by 40-70%. Covers embedding similarity, similarity thresholds, GPTCache, Redis, invalidation strategies, and real cost math.
Every production LLM app eventually hits the same problem. Traffic scales, the API bill climbs, and you notice something: a large chunk of your users are asking the same things, just worded differently.
"What's your return policy?" "How do I return something?" "Can I send this back?" Three queries. Three LLM calls. One answer.
Traditional caching can't help here. It matches on exact strings. Natural language almost never repeats exactly. Semantic caching solves this by matching on meaning. If a new query is semantically close enough to a cached one, you return the stored response. The LLM never gets called.
Industry data puts 30-40% of LLM requests as semantically similar to previous ones. In high-traffic FAQ bots and internal knowledge tools, that number can exceed 60%. Leaving that on the table means paying full inference cost for questions your system already knows how to answer.
This guide covers how semantic caching works, how to implement it, how to tune the key parameters that make or break it, and what realistic cost savings look like across different app types.
What Semantic Caching Actually Does
A semantic cache sits between your application and your LLM. Every incoming query gets converted into a vector embedding. That embedding gets compared against embeddings stored from previous queries. If the similarity score clears a threshold, the cached response comes back. If not, the query goes to the LLM, and the new response gets stored for next time.
User query
↓
Embed query (5-20ms)
↓
Vector similarity search (5-30ms)
↓ similarity > threshold?
Yes → return cached response (total: 10-50ms)
No → call LLM (1-10s) → cache response → return
For a cache hit, total response time drops from 1-10 seconds to 10-50ms. That's the latency win. The cost win is simpler: zero tokens consumed.
How it differs from prefix caching
Prefix caching (what OpenAI, Anthropic, and DeepSeek do natively) is a different optimization. It stores the KV tensors computed during prompt processing so that subsequent requests sharing the same prefix can skip recomputing that part. Anthropic's prompt caching, for instance, gives a 90% discount on cached input tokens and reduces latency by up to 85% for long prompts.
Semantic caching operates at the application layer and catches repeated intent across users. Prefix caching operates inside the model itself and speeds up computation for repeated prefixes. They complement each other. A well-architected system uses both: semantic caching catches repeated questions, prefix caching reduces cost on everything that gets through.
The Architecture
Four components make up any semantic cache:
1. Embedding model: Converts each query to a vector. The choice of model affects hit rate and accuracy. General-purpose models like text-embedding-3-small or all-MiniLM-L6-v2 work for most use cases. Domain-specific corpora benefit from models fine-tuned on similar text. The embedding model selection guide covers this tradeoff in detail.
2. Vector store: Holds the query embeddings and their corresponding responses. Redis with vector search, FAISS, Qdrant, and Milvus are all commonly used. The choice depends on your latency requirements, scale, and whether you need persistence.
3. Similarity search: Takes the incoming embedding and finds the closest stored embedding using cosine similarity or inner product. Approximate nearest neighbor (ANN) algorithms like HNSW keep this fast even at large cache sizes.
4. Threshold logic: The gate that decides whether a match is close enough to reuse. This is the parameter that makes or breaks the system.
GPTCache: The Standard Starting Point
GPTCache is the most widely used open-source semantic caching library for LLMs. It integrates with LangChain and LlamaIndex and supports multiple backends.
from gptcache import cache
from gptcache.adapter import openai
from gptcache.embedding import Onnx
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation
# Initialize embedding model
onnx = Onnx()
# Set up storage: SQLite for responses, FAISS for vectors
data_manager = get_data_manager(
CacheBase("sqlite"),
VectorBase("faiss", dimension=onnx.dimension)
)
# Configure cache
cache.init(
embedding_func=onnx.to_embeddings,
data_manager=data_manager,
similarity_evaluation=SearchDistanceEvaluation(),
)
# Drop-in replacement for the OpenAI client
cache.set_openai_key()
After this, your existing OpenAI calls run through the cache automatically. No other code changes needed.
GPTCache's default similarity evaluation uses cosine distance. A distance of 0 is identical, 1 is orthogonal. Most teams start with a threshold around 0.15-0.25 distance (equivalent to 0.75-0.85 cosine similarity) and tune from there based on their false positive rate.
One practical limitation: GPTCache uses SQLite by default, which doesn't scale well under concurrent write load. For production, swap the cache backend to Redis or PostgreSQL.
Redis Semantic Cache with LangChain
Redis is the most common production backend for semantic caching. It keeps embeddings and responses in memory, supports persistence, and has native vector search capabilities.
from langchain.globals import set_llm_cache
from langchain_community.cache import RedisSemanticCache
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
# Connect semantic cache to Redis
set_llm_cache(
RedisSemanticCache(
redis_url="redis://localhost:6379",
embedding=OpenAIEmbeddings(),
score_threshold=0.90, # cosine similarity threshold
)
)
# All subsequent LLM calls go through the cache
llm = ChatOpenAI(model="gpt-4o", temperature=0)
response = llm.invoke("What is the company refund policy?")
# Second call with same intent returns instantly from cache
response2 = llm.invoke("How do I get a refund?")
For distributed deployments, Redis Cluster distributes the cache across multiple nodes. A multilayer approach often works well in practice:
# Tier 1: in-memory exact match (sub-ms, handles rapid repeats)
# Tier 2: Redis semantic cache (10-50ms, handles paraphrases)
# Tier 3: LLM call (1-10s, genuinely new queries)
from langchain.cache import InMemoryCache, RedisSemanticCache
# In production, implement both layers explicitly
# using a custom cache manager that checks in-memory first
AWS MemoryDB and Amazon ElastiCache for Redis both support vector search and work as drop-in replacements for self-hosted Redis in cloud deployments. For self-hosted vector search at scale, Qdrant and Milvus add more control over indexing strategy and ANN parameters.
The Threshold Problem: Most Teams Get This Wrong
The similarity threshold is the most critical parameter in your semantic cache. It controls the precision-recall tradeoff for cache hits. Set it wrong and you either miss most of your savings (too strict) or serve wrong answers (too loose).
At 0.85: You catch most paraphrases but risk false positives. "What time is the store open?" and "When does the store close?" can hit cosine similarity of 0.85. Those need different answers. Threshold at 0.85 would return the wrong cached response.
At 0.95: Much safer. You only hit the cache when queries are very close rephrasings. But you'll miss a lot of legitimate savings from more varied phrasings of the same question.
At 0.90: The recommended default for most applications. A Dataquest study on a realistic workload found a 40.9% cache hit rate at 0.90 threshold, versus near-zero false positives.
The problem with a single global threshold is that different query categories have different "density" in embedding space. Research published in late 2025 found:
| Query type | Hit rate | Recommended threshold |
|---|---|---|
| Code/documentation queries | 40-60% | 0.88 (dense cluster, more permissive) |
| FAQ / help desk queries | 30-50% | 0.92 (medium density) |
| Conversational queries | 5-15% | 0.95 (sparse, strict threshold) |
| Time-sensitive queries | Not cacheable | Skip |
| Personalized queries | Not cacheable | Skip |
Code queries cluster tightly in embedding space because they use constrained vocabulary (function names, libraries, APIs). Conversational queries distribute sparsely because people phrase open-ended questions in highly varied ways. A single threshold performs poorly across both.
How to tune your threshold
The right approach is empirical, not guesswork:
- Log 500-1,000 real query pairs at varying similarity scores from your production traffic
- Label each pair as "same intent" or "different intent" (automated LLM labeling works here if you verify a sample)
- Compute precision at each threshold level
- Set the threshold at the point where precision exceeds 97-98% for your acceptable false positive rate
A VentureBeat analysis found that at a 67% cache hit rate, the 20ms overhead for embedding and vector search was negligible compared to the 850ms LLM call avoided. The net latency improvement was 65% alongside the cost reduction. Even at p99, the 47ms vector search overhead stays acceptable.
Cache Invalidation Strategies
A cache that serves stale responses erodes user trust faster than a slow app. You need a clear invalidation strategy before you go live.
TTL (Time-to-Live): The simplest approach. Set an expiry on each cached entry. Short TTLs (hours) for time-sensitive content, longer TTLs (days or weeks) for stable knowledge base content.
# Redis TTL example
cache.setex(
f"cache:{query_hash}",
86400, # 24 hours
json.dumps({"response": response, "embedding": embedding.tolist()})
)
Add random jitter to TTLs to avoid cache stampedes when many entries expire simultaneously:
import random
ttl_jitter = random.randint(0, 3600) # up to 1 hour of jitter
cache.setex(key, base_ttl + ttl_jitter, value)
Event-driven invalidation: When your underlying data changes, publish an invalidation event. Product price updated, policy document revised, or knowledge base entry changed: all of these should trigger targeted cache invalidation, not a full flush.
# On data update, publish invalidation event
redis_client.publish("cache-invalidation", json.dumps({
"topic": "refund_policy",
"invalidate_tags": ["refund", "return", "policy"]
}))
# Cache consumer listens and invalidates matching entries
def on_invalidation(message):
tags = json.loads(message["data"])["invalidate_tags"]
for tag in tags:
keys = redis_client.smembers(f"tag:{tag}")
for key in keys:
redis_client.delete(key)
Versioning: Tag cached responses with a content version. When source content is updated, increment the version. Lookups check version before serving cached content.
Freshness checks: For content that can go stale without an explicit event (news, stock prices, regulatory information), run periodic freshness verification on a sample of cached entries. The VentureBeat analysis found this catches staleness that TTL and event-based invalidation miss. This is especially relevant for teams using continual learning to keep models updated, where the underlying model changing makes previously cached responses unreliable.
Queries that should never be cached:
- Responses containing user-specific data (account balances, order history)
- Time-sensitive responses ("what time is it", "current stock price")
- Transactional confirmations (order placed, password reset initiated)
- Responses that depend on user state or session context
def should_cache(query: str, response: str, context: dict) -> bool:
# Never cache time-sensitive queries
time_patterns = ["current", "now", "today", "latest", "right now"]
if any(p in query.lower() for p in time_patterns):
return False
# Never cache personalized responses
if context.get("user_id") and "{user_name}" in response:
return False
# Never cache very short responses (likely errors or low-value)
if len(response.split()) < 10:
return False
return True
What Hit Rates Look Like in Practice
Hit rate is the percentage of queries that return from cache. It's the primary driver of cost savings. Hit rate varies dramatically by application type.
| Application type | Typical hit rate | Notes |
|---|---|---|
| Internal HR/policy chatbot | 50-70% | Employees ask similar questions repeatedly |
| Customer FAQ bot | 40-60% | High query repetition across users |
| Code assistant (docs/patterns) | 40-60% | Dense query clusters |
| General-purpose assistant | 10-30% | High query diversity |
| Research/analytical queries | 5-15% | Low repetition by nature |
| Personalized AI assistant | 0-10% | Context-dependent, mostly not cacheable |
A 2025 research paper on category-aware caching found that high-repetition categories (code, documentation, FAQ) hit 40-60%, while conversational categories hit 5-15%. Vector database lookup costs (30ms) require a break-even hit rate of 15-20% to be worth running the cache at all. Low-repetition workloads may not justify the added infrastructure complexity.
Real-world example: customer support bot
Assumptions:
- 100,000 queries/month
- Average query: 50 input tokens, 200 output tokens
- Model: GPT-4o at $2.50 input / $10.00 output per million tokens
- 45% semantic cache hit rate
Without caching:
Input cost: 100,000 × 50 × ($2.50/1,000,000) = $12.50
Output cost: 100,000 × 200 × ($10.00/1,000,000) = $200.00
Total: $212.50/month
With 45% semantic cache hit rate:
Cache hits (45,000 queries): $0 LLM cost
Cache misses (55,000 queries):
Input cost: 55,000 × 50 × ($2.50/1,000,000) = $6.88
Output cost: 55,000 × 200 × ($10.00/1,000,000) = $110.00
Total: $116.88/month
Monthly savings: $95.62 (45%)
Additional infrastructure cost: Redis at ~$10-30/month for this volume. Net savings still significant.
At 60% hit rate (internal knowledge base), savings jump to 60% of the original bill. At 70%:
Cache misses (30,000 queries): $3.75 + $60.00 = $63.75
Savings: $148.75/month (70%)
These numbers assume you're only counting LLM API cost. Add in reduced latency improving conversion rates, fewer timeout errors under traffic spikes, and consistent response quality on repeated queries, and the case gets stronger. For a more complete breakdown of cost reduction strategies across the full stack, the LLM cost optimization guide covers caching alongside model selection, batching, and quantization.
Two-Tier Caching: Exact Match First, Then Semantic
An often-overlooked optimization: run exact-match caching as a first layer before semantic caching. Exact matches are dictionary lookups taking under 1ms. They catch the most common case (same user hitting the same query twice) at nearly zero cost. Only queries that miss the exact cache go through embedding generation and vector search.
from langchain.cache import InMemoryCache
import hashlib
class TwoTierCache:
def __init__(self, semantic_cache, exact_cache_size=1000):
self.exact_cache = {} # or use LRU cache
self.semantic_cache = semantic_cache
self.hits_exact = 0
self.hits_semantic = 0
self.misses = 0
def get(self, query: str):
# Layer 1: exact match (sub-ms)
query_hash = hashlib.sha256(query.encode()).hexdigest()
if query_hash in self.exact_cache:
self.hits_exact += 1
return self.exact_cache[query_hash]
# Layer 2: semantic match (10-50ms)
result = self.semantic_cache.lookup(query)
if result:
self.hits_semantic += 1
# Also add to exact cache for future identical queries
self.exact_cache[query_hash] = result
return result
self.misses += 1
return None
def set(self, query: str, response: str):
query_hash = hashlib.sha256(query.encode()).hexdigest()
self.exact_cache[query_hash] = response
self.semantic_cache.store(query, response)
def hit_rate_report(self):
total = self.hits_exact + self.hits_semantic + self.misses
return {
"exact_hit_rate": self.hits_exact / total,
"semantic_hit_rate": self.hits_semantic / total,
"overall_hit_rate": (self.hits_exact + self.hits_semantic) / total,
"miss_rate": self.misses / total,
}
This matches what research consistently finds: exact-match caching at Layer 1 is effectively free, and the semantic layer only pays the embedding overhead for genuinely novel queries.
The Adaptation Tier: Going Beyond Binary Cache/Miss
Standard semantic caching is binary: either similarity is above threshold (cache hit) or it's not (full LLM call). An adaptation tier adds a middle option.
When similarity falls in a range like 0.70-0.85, the cached response is close but not identical. A cheaper model can adapt it to the new query. One team's implementation found this handled 35% of their queries at a fraction of the full generation cost:
DIRECT_HIT_THRESHOLD = 0.85
ADAPT_THRESHOLD = 0.70
def get_response(query: str, llm_full, llm_cheap) -> str:
results = vector_store.search(query_embedding, k=1)
if not results:
# Full miss
response = llm_full.invoke(query)
cache_store(query, response)
return response
similarity = results[0].score
cached_response = results[0].payload["response"]
if similarity >= DIRECT_HIT_THRESHOLD:
# Direct hit: return as-is
return cached_response
elif similarity >= ADAPT_THRESHOLD:
# Near hit: cheap adaptation
# Cost: ~$0.002 vs ~$0.015 for full generation
adapted = llm_cheap.invoke(
f"Original question: {results[0].payload['query']}\n"
f"Original answer: {cached_response}\n"
f"New question: {query}\n"
f"Adapt the answer for the new question. Keep it concise."
)
return adapted
else:
# Full miss
response = llm_full.invoke(query)
cache_store(query, response)
return response
On 100 queries with 10 topics and 10 paraphrases each, this architecture reduced cost by 72% while maintaining response quality. The adaptation tier is where most of the savings came from. Full misses were rare once the cache had reasonable coverage.
The overhead of the adaptation call (using Claude Haiku or GPT-4o-mini) costs around $0.002 versus $0.015 for full generation on a typical query. That's an 87% cost reduction on near-hit queries.
Security: The Threat Nobody Talks About
Semantic caches introduce a security surface that exact-match caches don't have. The 2025 research calls it cache poisoning.
An attacker can craft a query that embeds close to a legitimate query in vector space. When that attacker's query gets answered and cached, the poisoned response becomes the cached result for legitimate users asking the real question.
For multi-tenant applications, this is a real risk. A malicious query designed to look like "How do I reset my password?" can cache a response with a phishing link, which then gets served to legitimate users asking the same question. For enterprises running sensitive workloads, the private AI platform guide covers how isolated deployment architectures eliminate cross-tenant cache exposure entirely.
Practical mitigations:
- Never share a global semantic cache across different organizations or privilege levels
- Validate cached responses against a list of disallowed patterns before serving
- Log false positives (cases where the cache served a wrong response) and monitor the rate
- Consider re-ranking cached responses with a lightweight judge model before serving in high-stakes contexts
- For regulated or security-sensitive deployments, set a tighter threshold or disable caching for sensitive query categories
The 0.8% false positive rate observed in well-tuned production systems is within acceptable bounds for most FAQ applications but would be unacceptable for medical advice or financial transaction flows. For enterprises where response accuracy is a compliance requirement, the LLM reliability and evaluation guide covers how to build systematic quality checks on top of your caching layer.
Combining Semantic Caching with Provider Prefix Caching
The maximum savings come from layering both strategies:
| Layer | What it catches | Savings |
|---|---|---|
| Semantic cache hit | Same question, different wording | 100% of token cost |
| Provider prefix cache hit | Same system prompt/context prefix | 50-90% of input token cost |
| Both miss | Genuinely new query with new context | Baseline cost |
If your RAG pipeline loads documents into a prompt for context, those document tokens repeat across many queries. Provider prefix caching (Anthropic, OpenAI, Gemini, DeepSeek all support this) cuts the cost of those repeated tokens. Semantic caching cuts the cost of repeated questions.
For a RAG system with a 2,000-token system prompt and retrieved context:
Without any caching:
2,000 input tokens × $3.00/M = $0.006 per query
With semantic caching (60% hit rate):
60% of queries: $0 (full cache hit)
40% of queries:
With prefix caching: 2,000 input tokens at 10% = $0.0006
Without prefix caching: $0.006
Combined savings:
60% from semantic: $0
40% × 90% from prefix: $0.0006
vs $0.006 baseline = 90% reduction on input tokens that do get processed
For teams on PremAI running self-hosted models, semantic caching reduces GPU utilization and inference costs independently of provider pricing. The inference cost optimization guide covers the full cost stack.
Monitoring: What to Track
A semantic cache with no monitoring is a liability. You can't tell if it's working, and you won't know when it starts serving wrong answers.
The metrics that matter:
class CacheMetrics:
def record_hit(self, query, similarity, latency_ms):
self.metrics.counter("cache.hit").increment()
self.metrics.histogram("cache.similarity_score").observe(similarity)
self.metrics.histogram("cache.hit_latency_ms").observe(latency_ms)
def record_miss(self, query, llm_latency_ms, tokens_used):
self.metrics.counter("cache.miss").increment()
self.metrics.histogram("cache.miss_latency_ms").observe(llm_latency_ms)
self.metrics.counter("tokens.consumed").increment(tokens_used)
def record_false_positive(self, query, cached_query, similarity):
# Triggered when user feedback or judge model flags wrong response
self.metrics.counter("cache.false_positive").increment()
self.metrics.histogram("cache.false_positive_similarity").observe(similarity)
Key thresholds to alert on:
- False positive rate above 1%: threshold is probably too low, or cache is growing stale
- Hit rate below 15%: embedding overhead may exceed the savings. Evaluate if caching is worth running
- Vector search p99 above 100ms: index may be too large or ANN parameters need tuning
- Cache size growing without bound: eviction policy may not be working
For teams using PremAI's observability stack, the LLM observability guide covers integrating cost and latency metrics across the full inference pipeline. The evaluations overview shows how to set up automated response quality checks that can flag cache false positives.
Semantic Caching for RAG Pipelines
Semantic caching in a RAG system is slightly different from caching standalone LLM calls. There are two cacheable layers:
Query-level caching: Cache the final LLM response for the full query. This is what we've covered so far. If a new query matches a cached one closely enough, return the stored response without retrieval or generation.
Retrieval-level caching: Cache the retrieved document chunks for a query. If a new query closely matches a previously answered one, reuse the same retrieved context before calling the LLM. This saves the retrieval step but still pays for generation.
Query → Semantic cache check
↓ miss
Retrieval cache check (skip vector DB lookup if similar query cached)
↓ miss
Full retrieval + LLM generation → cache both
Query-level caching gives bigger savings (100% cost reduction on hits). Retrieval-level caching is a useful fallback when the same question gets slight variations that change the ideal response but share the same relevant documents.
For the full architectural picture of where caching fits in a production RAG system, the advanced RAG methods guide covers hybrid retrieval and optimization techniques. The RAG strategies overview covers how caching integrates with the broader pipeline. If your RAG system processes sensitive documents, the RAG privacy guide is worth reading before implementing a shared cache layer.
When Semantic Caching Is Not Worth It
Not every LLM application benefits from semantic caching. Before building it, check the break-even math.
Vector search on a remote Redis instance adds 20-50ms per request. At p99, you might see 100ms. That overhead only pays off if your hit rate exceeds 15-20%. Below that, you're adding latency to every request while saving on only a small fraction.
Skip semantic caching if:
- Your queries are highly personalized or context-dependent
- Your responses contain real-time data that changes frequently
- Your query volume is low enough that API costs aren't a meaningful budget item
- Your use case is agentic (multi-step reasoning where intermediate outputs vary per run)
Consider a lighter approach instead: just enable prefix caching at the provider level. For Anthropic, that's a cache_control parameter on your messages. For OpenAI, it's automatic. No infrastructure to maintain, no threshold to tune, 50-90% savings on repeated prefixes.
Use semantic caching on top of prefix caching when your traffic shows repetitive query patterns and your hit rate projections exceed the break-even threshold.
Implementation Checklist
Before deploying to production:
Threshold tuning
- [ ] Collect 500+ real query pairs with similarity scores
- [ ] Label same-intent vs different-intent pairs
- [ ] Set threshold at precision > 97% on your labeled set
- [ ] Use category-aware thresholds if your queries span multiple types
Storage
- [ ] Use Redis, Qdrant, or Milvus (not SQLite) for production
- [ ] Configure an eviction policy (LRU is standard)
- [ ] Set appropriate index size limits based on your memory budget
Invalidation
- [ ] Define TTL policy per content type
- [ ] Add TTL jitter to prevent stampedes
- [ ] Implement event-driven invalidation for mutable content
- [ ] Build an exclusion list for query types that should never be cached
Monitoring
- [ ] Track hit rate, false positive rate, and p99 vector search latency
- [ ] Alert when false positive rate exceeds 1%
- [ ] Log similarity scores on hits for ongoing threshold calibration
Security
- [ ] Isolate caches per tenant/org if multi-tenant
- [ ] Validate cached responses before serving in sensitive contexts
- [ ] Monitor for cache poisoning patterns in high-stakes flows
FAQ
How do I know what hit rate to expect before building?
Log a sample of your real traffic and cluster queries by semantic similarity. If 40%+ of your query pairs fall within cosine similarity 0.88-0.95, you have a cacheable workload. If the distribution is flat (most queries unique), caching won't help much. Customer support bots and internal knowledge tools almost always cache well. Open-ended creative or analytical assistants usually don't.
Does semantic caching work with streaming responses?
Yes, with a small tradeoff. The simplest approach is stream-then-cache: stream the full response to the user, then store it in the cache once generation is complete. For cache hits, return the full cached response immediately (not streamed), which is actually faster than streaming. If you need the visual streaming experience on cache hits, you can simulate it by "streaming" the cached string with artificial delays.
How should I handle multi-turn conversations?
Single-query semantic caching treats each message independently. Follow-up questions like "What about Kubernetes instead?" need the prior context to make sense. The solution is to include a representation of the conversation context in the cache key, or to use a context-aware caching library like MeanCache. Most production deployments start with single-turn caching and add context awareness after validating the basic system works.
Can I fine-tune the embedding model to improve hit rates?
Yes, and it helps significantly. Domain-specific embedding models recognize paraphrases that general-purpose models miss. A 2025 paper on domain-specific embeddings found that fine-tuning on your actual query distribution improves hit rate by meaningful amounts without changing the threshold. The enterprise fine-tuning guide covers the dataset and training setup. PremAI's platform handles fine-tuning and evaluation for embedding models alongside generative models, which matters if you want to validate cache quality systematically using the evaluations framework.
What's the performance overhead of the embedding step on every query?
Running a lightweight embedding model like all-MiniLM-L6-v2 locally adds 2-5ms per query. Using an API-hosted embedding model (OpenAI, Voyage) adds 10-30ms. For cache hits, this overhead is negligible: you're still returning in under 50ms versus 1-10 seconds for LLM inference. For cache misses, you pay the embedding overhead plus normal LLM latency. At a 40% hit rate, the math strongly favors running the cache. For self-hosted inference setups, running the embedding model locally eliminates the API roundtrip entirely.
Need to reduce LLM inference costs on self-hosted models while maintaining data sovereignty? PremAI's platform handles the full inference stack including caching, evaluation, and fine-tuning. Talk to the team to see how it applies to your setup.