AI system design interview: design a document extraction pipeline
Run extraction as an asynchronous pipeline: intake, classification, parsing with OCR for scans, schema-constrained extraction, validation with business rules, human review for anything that fails or has low confidence, then export with an audit log. Trust the rules, not the model: line items must sum to the total, the supplier must exist, the tax rate must be valid. Reviewer corrections become your eval set and, later, training data.
Question
Extract fields from 50,000 supplier invoices a month (PDFs, scans, emails) into an ERP.
Clarify
- Fields needed: header fields, line items, tax, bank details.
- Number of suppliers and layouts.
- Share of scanned vs digital documents.
- Required accuracy and how errors are caught today.
- Turnaround time: same day, or within minutes?
Estimate
50,000 invoices a month is about 2,300 per working day. If processing one takes 20 seconds, that's under 13 hours of sequential work a day, so a handful of parallel workers clears the queue in an hour.
Architecture
| Stage | Component |
|---|---|
| Intake | Email inbox and upload, deduplication by file hash |
| Queue | Job queue with workers and retries |
| Classify | Invoice, credit note, receipt, other |
| Parse | Text extraction for digital PDFs, OCR for scans, tables kept as tables |
| Extract | Structured output against a schema, per document type |
| Validate | Sum checks, supplier lookup, tax rules, duplicate invoice numbers |
| Review | UI showing flagged fields with the source region highlighted |
| Export | ERP integration, audit log of every value and who approved it |
Deep dive: confidence and review
Fields that fail validation or have low confidence go to a reviewer, everything else goes straight through. Start with a strict threshold, so most documents get reviewed, and loosen it as measured accuracy improves. Every correction is stored with the original value, which gives you a growing labelled set for evals.
Failure modes
| Failure | Mitigation |
|---|---|
| Tables parsed wrong | Layout-aware parsing, line-item sum check |
| Model "corrects" a total that doesn't add up | Extract what's printed, let rules flag the mismatch |
| Duplicate invoices | Hash and invoice number checks |
| New supplier layouts | Monitor accuracy per supplier, add examples |
| Bank detail changes (fraud risk) | Always route to review |
Metrics
Field-level accuracy, straight-through rate, review time per document, and cost per document.
What interviewers listen for
Validation with rules, not trust in the model, and a feedback loop from reviewers.
From the floor
In the webinar on AI agents for business leaders, I named financial operations as the second quick win after customer service, invoice processing, reconciliation and fraud detection, because those tasks are slow, error-prone and high-risk when done by hand, and a system that tracks and logs everything reduces that risk.
Related
- Document processing reference
- Document parsing
- Async job queue pattern
- Structured outputs and function calling