Data architecture for AI systems
An AI system needs more than a vector database. It has source systems, an ingestion pipeline, a document store with clean text and metadata, vector and keyword indexes, relational databases for exact values, memory stores, a trace store, and versioned eval and fine-tuning datasets, with lineage linking every answer back to the source version it came from. Lineage is what makes AI answers auditable and fixable: when a user disputes an answer, you can say which version of which document produced it.
Why a vector database isn't the whole story
Tutorials show a PDF, a vector database and a model. Production systems have many sources that change, permissions that matter, values that must be exact, users whose data must be deletable, and a team that needs to know why an answer was given. Each of those needs a place in the data architecture.
Stores
| Store | Holds | Notes |
|---|---|---|
| Sources | Original systems of record: drives, wikis, CRM, ERP, databases | You don't own their schedules or formats |
| Raw store | Original files as ingested | For reprocessing when parsing improves |
| Document store | Parsed text, metadata, versions, permissions | The clean layer everything else is built from |
| Indexes | Vectors, keywords, optional entity graph | Rebuilt from the document store |
| Relational data | Exact values for tools and SQL | The source of truth for numbers |
| Cache | Answers and embeddings | Scoped and expiring |
| Memory | User and agent memory | Keyed by tenant and user, deletable |
| Traces | Requests, steps, outputs, feedback | Masked, with retention limits |
| Datasets | Eval sets, fine-tuning sets | Versioned, with sources |
Flow
sources ──> ingestion (pull, parse, clean, mask, chunk, enrich)
│
├──> raw store (originals)
└──> document store (text, metadata, version, permissions)
│
├──> vector index
├──> keyword index
└──> entity graph
query time: indexes + relational data (tools) ──> model ──> answer
│
traces ──> evals, datasets, feedback
Raw, processed and gold
A useful folder or bucket structure for any data pipeline:
- Raw: exactly what came in, never edited.
- Processed: cleaned, normalised, chunked.
- Gold: labelled and approved data, used for evals and training.
When parsing or chunking improves, you reprocess from raw instead of fetching everything again.
Lineage
Every chunk should carry its source document ID, the document version, and the ingestion run that produced it. Every answer's trace should record which chunks and which tool results it used. Then "why did the bot say this?" has an answer: this chunk, from this version of this document, ingested on this date.
Freshness and deletes
- Decide how fresh each source must be (policy changes within an hour, archive documents weekly), and schedule ingestion to match.
- When a document is deleted or a permission changes at the source, propagate it to the document store, every index, the cache and any datasets.
- When a user asks to be deleted, find their data in memory, traces, caches and datasets, not only in the main database.
Ownership and retention
For each store, write down its owner, its retention period, who can read it, and how deletes propagate. That table is what a privacy review or an auditor will ask for.
From the floor
In the legal LLM project lab, the data pipeline used exactly this structure: a data folder with raw, processed and gold, ingestion from public sources like a contracts dataset and the text of India's Digital Personal Data Protection Act, normalisation, splitting by article, section and clause, and labelling, with a data card describing the sources, all before any model was trained.
In the production pipeline I teach, retrieval has an offline side, ingesting the company's raw unstructured data, chunking it, embedding it and storing it in a vector database, with a graph database alongside it, and an online side that searches it at query time, while order data and other exact values come from the operational database through tools.