Data for AI: pipelines, quality, labelling, synthetic data, feedback and versioning
Better data usually helps an AI system more than a better model does: the better your data, the better your retrieval, your evals, your fine-tuned models and your answers. The data work behind an AI system has six parts: pipelines that keep content current, quality checks before retrieval, labelling by people who know the domain, synthetic data where real data is missing, feedback collected from real use, and versioning so every result can be reproduced. None of it is glamorous, and most production problems trace back to one of them.
Why data decides quality
When a RAG answer is wrong, the cause is often the data: the policy in the index is the old version, a table was parsed into nonsense, three versions of the same document disagree. When a fine-tuned model behaves badly, the cause is often the labels. When an eval says the system improved and users say it didn't, the eval set is often stale.
A better model can't fix any of these. Fixing the data can.
Where data flows in an AI system
sources ──> pipelines ──> clean documents ──> indexes ──> answers
│ │
└── quality checks ▼
traces + feedback
│
labelling by experts <── sampled traces <──────────────────┘
│
├──> eval sets (versioned)
└──> fine-tuning data (versioned, plus synthetic where needed)
What this section covers
| Page | The question it answers |
|---|---|
| Data pipelines | How do I keep the AI's knowledge current? |
| Data quality for retrieval | What should I fix in the corpus before tuning RAG? |
| Labelling and annotation | Who labels, and how do I keep labels consistent? |
| Synthetic data | When can generated data replace real data? |
| Feedback collection | Which signals should the product capture? |
| Data versioning | How do I reproduce any result? |
From the floor
In the legal LLM project lab, the data strategy was the first phase, before any model: public sources pulled into a raw folder, normalised and split by article, section and clause into a processed folder, labelled in Label Studio into a gold folder, with a data card describing every source, because every company labels and prepares its data first, and the better the data is, the better the model will be.