AI system design interview: design LLM observability for a team
Pick one tracing standard (OpenTelemetry with its GenAI conventions) and put it in a shared library, so every product emits the same spans for model calls, retrieval, tools and agent steps. Each model span records the rendered prompt, model and prompt version, tokens, latency, cost and finish reason, with PII masked before export. Keep all error traces, sample the rest for retention, link user feedback by trace ID, and show evaluator results next to the traces they scored.
Question
20 engineers ship LLM features across five products. Design the observability setup.
Clarify
- Request volume per product.
- Data sensitivity and where traces may be stored.
- Existing observability stack (logs, metrics, APM).
- Who reads traces: engineers, product managers, domain experts?
What each span records
| Span | Fields |
|---|---|
| Model call | Rendered prompt, model, prompt version, input, output and cached tokens, latency, time to first token, cost, finish reason |
| Retrieval | Query, filters, returned document IDs and scores |
| Tool call | Tool name, arguments, result summary, errors |
| Agent step | Step number, decision, state |
| Request | User and tenant (pseudonymised), feature, outcome |
Architecture
1 - Shared library wraps model, retrieval and tool calls and emits spans. 2 - PII masking runs before export. 3 - Spans go to an LLM-aware backend (Langfuse, Phoenix, LangSmith or similar), and metrics also go to the existing dashboards. 4 - Retention: all error and flagged traces kept, a sample of the rest. 5 - Feedback events (thumbs, edits, escalations) carry the trace ID. 6 - Evaluators run on sampled traces and write scores back to them.
Dashboards and alerts
Latency, cost and error rate per product and model, token usage per feature, and evaluator scores over time, with alerts on cost spikes, error rate and score drops.
Failure modes
| Failure | Mitigation |
|---|---|
| Each team instruments differently | One shared library, reviewed spans |
| PII stored in traces | Masking before export, scans |
| Trace volume too expensive | Sampling with all errors kept |
| Nobody reads traces | A weekly review of sampled traces per product |
What interviewers listen for
One standard across teams, prompt and version in every span, privacy in the pipeline, and traces connected to feedback and evals.
From the floor
In the DevFix agent build, the agent runs code, checks errors and fixes them in a loop, and with a loop like that, you can't debug without seeing every step, the input, the tool result and the decision, which is what a trace gives you.