AI system design interview: design multi-tenant RAG for a SaaS product
Isolate by tenant in every component that stores or reuses data: vector namespaces or mandatory tenant filters, the tenant in every cache key, tenant-scoped memory, and tenant tags on traces with access control. Add per-tenant configuration, rate limits and spend caps, separate ingestion queues so one large upload doesn't slow everyone, and cross-tenant leak tests in CI.
Question
Your SaaS product lets each customer upload documents and ask questions. Design it for 2,000 tenants.
Clarify
- Size distribution: a few large tenants and many small ones is typical.
- Isolation requirements: contractual, regulatory, or both.
- Whether large customers need their own models, regions or encryption keys.
Isolation options
| Option | Isolation | Cost and operations |
|---|---|---|
| Shared index with mandatory tenant filter | Logical, depends on every query applying the filter | Cheapest, simplest |
| Namespace or collection per tenant | Stronger, filter can't be forgotten | More objects to manage |
| Dedicated index or database per tenant | Strongest | Expensive at 2,000 tenants, fine for a few large ones |
A common answer is a mix: namespaces for most tenants, dedicated infrastructure for the few who pay for it.
Every shared component, checked
| Component | Leak risk | Control |
|---|---|---|
| Vector index | Retrieval returns another tenant's chunks | Namespace or enforced filter |
| Cache | A cached answer served to another tenant | Tenant in the key and in the cache index |
| Memory | Facts from one tenant's users recalled for another | Memory keyed by tenant and user |
| Traces and logs | Support staff see another tenant's data | Tenant tags, access control on the trace store |
| Fine-tuned adapters | One tenant's data in another's model | Per-tenant adapters only |
| Evaluation data | Mixed across tenants | Separate sets, permission to use data |
Noisy neighbours
Per-tenant rate limits and spend caps, ingestion queues per tenant or weighted fair scheduling, and per-tenant cost tracking so pricing reflects usage.
Leak testing
Seed two test tenants with distinct, recognisable documents and run queries from each that should only match the other's content, in CI and after every infrastructure change.
What interviewers listen for
Every shared component checked for leaks, and noisy-neighbour protection.
From the floor
In the HR query bot build, what the bot could retrieve depended on who was asking, through authenticated tokens that expire after 30 minutes, and multi-tenancy is the same principle applied one level up: the tenant is part of who is asking, in every store.
Related
- Multi-tenant AI platforms
- Metadata filtering and access control
- Semantic caching
- Access control for RAG and agents