AI system design interview: design a semantic cache
Use two layers: an exact-match cache keyed by a hash of the normalised query plus everything that changes the answer, then a semantic cache that searches embeddings of past queries in a scoped index and returns a stored answer only above a similarity threshold set from labelled pairs. Decide explicitly what's cacheable, invalidate entries when their source documents change, and track wrong hits, not just hit rate.
Question
Your assistant gets 1 million questions a day and many are repeats. Design a cache.
Clarify
- Which answers are user-specific or time-sensitive.
- How often the source content changes.
- What wrong-hit rate is acceptable, which depends on the cost of a wrong answer.
Estimate
If 30% of questions are exact or near repeats and the average call costs $0.004, caching them saves around $1,200 a day, and cache hits return in tens of milliseconds instead of seconds. The hit rate is the number to measure first, from your own logs, because these savings scale with it.
Architecture
1 - Normalise the query: trim, collapse whitespace, lowercase where safe. 2 - Exact layer: key is a hash of query, tenant, language, model and prompt version. 3 - Semantic layer: embed the query, search a cache index scoped by tenant and language, return the stored answer if similarity is above the threshold. 4 - Miss: call the full pipeline, then store the answer if it's cacheable, with the IDs of the source documents it used.
Deep dive: threshold and invalidation
Label 200 to 500 query pairs as same intent or different intent, compute their similarities, and choose the threshold that keeps wrong hits near zero. Invalidate an entry when any source document it used changes, and expire every entry with a TTL as a backstop.
What not to cache
- Answers that depend on the user's own data (orders, balances).
- Time-sensitive answers (stock, prices), unless the TTL is shorter than how often they change.
- Answers produced while a guardrail flagged the request.
Failure modes
| Failure | Mitigation |
|---|---|
| One user's answer served to another | Tenant and user scope in keys and index |
| Stale policy answer | Source-based invalidation, TTL |
| Near-identical questions with opposite meaning ("cancel" vs "can't cancel") | Threshold from labelled data, exact-match only for short queries |
| Cache poisoning by a bad answer | Store only answers that passed output checks, allow manual purge |
Metrics
Hit rate per layer, wrong-hit rate from sampled review, cost saved, and latency on hits.
What interviewers listen for
Scope in the key, an explicit rule for what's cacheable, and a threshold set from data.
From the floor
In the inference pipeline lecture, a builder asked whether the cache step at the gateway was prompt caching or semantic caching, and I explained it's semantic caching: at the gateway (LiteLLM in that design, Kong is another option), the query is turned into a query vector, which is the same thing as an embedding, the semantic cache is checked with it, and the request is only processed further on a miss. The gateway doesn't create the vector itself, it calls an embedding model for it.