What a production trace records, span by span
A trace is the full record of one user request moving through your system: a tree of spans, one for each piece of work (model calls, retrieval, tool calls, agent steps), each with its timing, inputs, outputs and errors. You can't evaluate what you can't see, so tracing comes before any eval. The most useful field of all is the exact prompt the model received, after every template was filled and every document was inserted.
Traces and spans
A trace follows one request from the moment it arrives to the moment the answer goes back. It's made of spans. Each span is one piece of work, with a start time, an end time, a parent span and attributes describing what happened.
In an LLM app, the spans that matter are:
- Model calls.
- Retrieval calls (embedding the query, searching the index, reranking).
- Tool calls.
- Agent steps, which contain the calls above.
The OpenTelemetry GenAI semantic conventions give these standard names, such as chat, embeddings, execute_tool and invoke_agent, so traces work across tools.
The conventions are still in development, so attribute names may change.
invoke_agent "refund request" 4.8s
├── chat planner 1.1s
├── retrieval "refund policy electronics" 0.3s
│ └── embeddings 0.1s
├── execute_tool get_order(ORD-123456) 2.0s error: timeout
├── execute_tool get_order(ORD-123456) 0.4s
└── chat final answer 1.0s finish_reason: stop
What to log on model spans
- The exact prompt that was sent, after templates were filled and documents inserted. Most bugs live in what the model saw, not in the template you think it saw.
- The output.
- The model name and version.
- Input tokens, output tokens and cost.
- Latency, including time to first token for streaming.
- The finish reason. A reply that stopped because it hit the length limit is a truncation bug, even if it looks like a quality problem.
- Settings like temperature and the tools offered.
What to log on retrieval spans
- The query sent to the index, which may differ from the user's words after rewriting.
- The chunks that came back, with their scores and source IDs.
- Which chunks made it into the prompt after reranking and filtering.
Without this, you can never tell a retrieval failure (the right document wasn't found) from a generation failure (it was found and the model misused it).
What to log on tool spans
- The tool name and the arguments.
- The raw result, or a trimmed version for very large results.
- Any error, and how many attempts it took.
An empty result or a 500 error is an event worth flagging. Agents break on bad tool results more often than on hard reasoning.
What to log across the request
- A session ID and turn number on every span in a chat product. A failure on turn 6 is often caused by something the model accepted on turn 2.
- The user or tenant ID (hashed if needed), for debugging and access audits.
- The prompt version and app version.
- Feedback events on the same trace: thumbs down, retries, handoffs to a human, abandoned sessions. These are the cheapest failure signals you will ever get.
Mask personal data before it's stored, and give traces a retention period that matches your policies. See data leakage and PII.
Check your logging before writing evals
Open 10 random traces. If someone who has never seen your code can't tell what happened in each one, fix the logging first. Every eval you write later depends on this.
A real incident
In July 2025, Replit's AI agent wiped a production database with records on 1,206 executives during an explicit code freeze, in a trial run by SaaStr founder Jason Lemkin. The public story came from chat screenshots. A trace with tool-level spans, showing which command ran, in which environment, with which instruction in context, turns that kind of incident into a clear postmortem.
From the floor
In DevFix, our self-healing coding agent, observability meant tracing every step: every node of the LangGraph agent, every LLM call, every Docker execution, with latency and cost, because when anything happens in production, every part should be traced so it's easy to find the error. We started with LangSmith and later switched to Langfuse, because LangSmith isn't fully open source, while Langfuse is open source and can be self-hosted.
When a builder asked whether there was a tree view instead of reading every step by hand, the answer was the trace view itself: every run appears as a tree of the agent's nodes, so you can open the step where it went wrong.