Data flywheels: how AI products learn from production
A data flywheel is the loop where usage creates traces and feedback, you turn them into eval cases, better retrieval content, prompt fixes and fine-tuning examples, the product improves, and more usage follows. It only turns if you capture feedback linked to traces and someone reviews it on a schedule. Strong general models are available to everyone, so the part competitors can't copy is your workflow, your data and the loop that tells your system whether it got the answer right.
Why most AI products don't improve after launch
Most AI products collect logs nobody reads. The team ships, moves to the next feature, and the product stays as good as it was on launch day, while users' questions, documents and expectations move on.
A flywheel makes improvement a routine, not a project.
How the loop works
- Capture: traces plus feedback (thumbs, edits to drafts, escalations, outcomes like "ticket resolved" or "invoice approved").
- Review: a weekly error analysis on a sample of traces. See find failure modes in traces.
- Convert: failures become eval cases, corrections become examples, missing content becomes documents for retrieval.
- Improve: fix prompts, retrieval, tools, or fine-tune.
- Measure: evals and online metrics, before and after.
Feedback signals
| Signal | What it tells you | Caution |
|---|---|---|
| Thumbs up and down | Direct opinion | Few users click, and a thumbs down has many causes |
| Edits to AI drafts | Exactly what was wrong | Store the original and the edit |
| Regeneration | The first answer wasn't good enough | Sometimes users just want options |
| Escalation to a person | The AI couldn't handle it | Some cases should always escalate |
| Outcomes | Whether the task succeeded | Needs integration with business systems |
Edits and outcomes are usually the most valuable, because they show what good looked like. See feedback collection.
What each turn produces
- New eval cases, so the same failure is caught next time.
- New or corrected documents, when retrieval had nothing to find.
- Prompt changes, for behaviour the model can follow with clearer instructions.
- Examples for few-shot prompts or fine-tuning datasets, once you have enough.
- New guardrails, for failures that must never reach users.
Making it turn
Put a 60-minute weekly review on the calendar, with one required output: new eval cases added. Track:
- Eval cases added per month from production.
- Quality trend over releases.
- Share of user corrections that led to a change.
Data, consent and privacy
Feedback data is user data. Tell users how their feedback is used, mask personal data before it enters eval sets or training data, and respect contracts that limit how customer data can be used, especially in B2B products.
From the floor
In my AI field report session for leaders, two points connect to this page: building bigger and bigger general models is now done by a few well-funded companies, so the opportunity for most teams is in solving "boring" problems like document processing, internal workflows and compliance checks, and in law, healthcare and defence, the hard part is the verification loop, with a person checking the output, because those professions can't afford a wrong answer.