Data versioning for AI: datasets, indexes, prompts and models
Version everything that affects an output: eval sets, training data, the corpus snapshot behind each index, prompts, model versions and configuration.
Small datasets can live in Git as JSONL, and large ones need a data versioning tool (DVC, lakeFS, or versioned object storage with manifests).
Record the versions used in every eval run and every trace, because requirements.txt makes code reproducible, and you need the same for data.
Problem
Eval scores dropped between two runs. Did the model change, the prompt, the eval set, the embedding model, the chunk size, or the documents in the index? If nobody recorded the versions, nobody knows, and the team spends a week guessing.
What to version
| Artifact | How |
|---|---|
| Eval sets | Versioned JSONL in Git, with a changelog |
| Fine-tuning data | Versioned files or a data versioning tool, with sources |
| Corpus snapshot | A manifest listing document IDs and versions at build time |
| Index | A build manifest: corpus snapshot, embedding model, chunking settings |
| Prompts | Files in the repository, with a version string |
| Models and adapters | A registry with versions and metadata (base model, data version, training settings) |
| Configuration | Model names, temperatures, top k, thresholds, in versioned config |
An index build manifest
import hashlib
import json
from datetime import date
def build_manifest(doc_versions: dict[str, int], embedding_model: str, chunk_size: int, overlap: int) -> dict:
snapshot = hashlib.sha256(json.dumps(doc_versions, sort_keys=True).encode()).hexdigest()[:12]
return {
"index_version": f"idx-{date(2026, 10, 16).isoformat()}-{snapshot}",
"corpus_snapshot": snapshot,
"documents": len(doc_versions),
"embedding_model": embedding_model,
"chunking": {"size": chunk_size, "overlap": overlap},
}
manifest = build_manifest({"refund-policy": 3, "shipping": 1}, "bge-small-en-v1.5", 1000, 200)
print(json.dumps(manifest, indent=2))
Store the manifest next to the index, and put its index_version on every trace and eval run.
Tools
- Git: code, prompts, configs and small datasets (up to a few megabytes).
- DVC: versions large files and datasets alongside Git, storing the data in object storage.
- lakeFS: Git-like branches and commits for data lakes.
- Versioned object storage (S3 versioning and similar) with manifests: simple and often enough.
- Model registries (MLflow, Weights & Biases, Hugging Face Hub): models and adapters with metadata.
Recording versions in runs
Every eval result should store: model version, prompt version, eval set version, index version, and config version. Every production trace should store the same. Then any result can be reproduced, and any regression traced to the change that caused it.
Measuring it
Can any eval result be reproduced from its recorded versions? Try it once a quarter.
From the floor
On day 5 of the 75-day challenge, I covered serialisation: when you train a model, it lives in memory, and you need to save it, its parameters, the weights and biases, in a known format so you can load it again for predictions, along with JSON and model registries, which is the same habit this page asks for with every artifact in an AI system.
For Model Smith, the plan used Weights & Biases as the system of record for the pipeline, with a model registry tracking each LoRA adapter's lineage back to the dataset and the training settings that produced it.