AI system design interview: design RAG over internal company documents
Treat ingestion and permissions as the core of the design: connectors with incremental sync, layout-aware parsing, structure-aware chunks with permission metadata, hybrid retrieval with filters applied inside the search, reranking, and answers with citations and an abstention path. Evaluate retrieval and generation separately, and measure how long a deleted document or a permission change takes to reach the index.
Question
Design a question answering system over 200,000 internal documents (wiki, drives, tickets) for 5,000 employees.
Clarify
- Document types and formats: PDFs with tables, slides, wiki pages, tickets.
- Update frequency and how fresh answers must be.
- Permission model: per document, per folder, per group.
- Languages.
- Whether data can leave the company network.
- Expected questions: single facts, procedures, comparisons across documents.
Estimate
200,000 documents at an average of 10 chunks each is 2 million chunks. At 1,024 dimensions in float32, that's about 8 GB of vectors before index overhead, which fits one vector database node, and the embedding cost for the initial load is a one-time batch job.
Architecture
| Stage | Component |
|---|---|
| Ingestion | Connectors with incremental sync and delete propagation |
| Parsing | Layout-aware parsing, tables kept as tables, OCR for scans |
| Chunking | By section, with section titles and document metadata |
| Indexing | Embeddings plus BM25, permission metadata synced from the source |
| Query | Query rewriting for follow-ups, hybrid retrieval with permission pre-filtering |
| Ranking | Cross-encoder reranking of the top candidates |
| Generation | Answer from context, citations, abstain below a threshold |
| Operations | Tracing, feedback, freshness lag monitoring |
Deep dive: permissions and freshness
Filtering happens in retrieval, never in the prompt, because a model can't reliably be told to ignore documents it has already seen. Permissions are synced from the source system and stored with each chunk, so a search returns only what the user can open. Deletes and permission changes must reach the index within a set time, measured as freshness lag, and tested.
Failure modes
| Failure | Mitigation |
|---|---|
| Stale versions answered | Version metadata, prefer latest, delete propagation |
| Exact terms missed (product codes, error IDs) | BM25 in hybrid search |
| Permission leak | Pre-filtering, leak tests in CI |
| Unfaithful summary of a correct chunk | Faithfulness checks on sampled answers |
| Unanswerable questions answered anyway | Abstention threshold, tested with unanswerable questions |
Evals and metrics
- Recall@k on labelled question and passage pairs.
- Faithfulness and abstention on sampled answers.
- Permission leak tests in CI, target zero.
- Adoption, repeat use and time saved, from feedback and usage.
What interviewers listen for
Ingestion and permissions treated as first-class, hybrid retrieval with a reason, and evaluation split into retrieval and generation.
From the floor
In the HR query bot build in AI Engineer HQ, access control was part of the design from the start: employees authenticate through OAuth2 with tokens that expire after 30 minutes, so what the bot can retrieve depends on who is asking, which is the same principle you apply to internal document search at company scale.
Related
- Retrieval and RAG
- Enterprise knowledge engine reference
- Metadata filtering and access control
- Data pipelines