Observability for Agents: What to Log Beyond the Trace

Most agent observability stops at the trace: the sequence of model calls and tool calls for one run. That's necessary, and on its own it answers questions about this run while leaving you unable to answer anything about your system.

Three layers are needed, and the two above the trace are where operational understanding actually lives.

Layer 1: the trace (get it complete)

Per run, per step. The parts most often missing:

• The exact rendered array sent on each call — not a template reference, not a rendered view. What the API received.
• Full tool results, pre-truncation, stored by reference if large. Debugging a wrong decision requires seeing what the agent saw, including the part your context limit cut.
• Timing per step, split into model time and tool time. These have different causes and different fixes, and one number for both hides which.
• Token counts per call, split into cached and uncached input plus output.
• Versions: model, prompt hash, tool definition versions.

⚠️ Storage costs are real. A retention policy of full fidelity for a short window, then sampled or summarized, works well — provided anomalous runs are exempt from downsampling, because those are the ones you'll want.

Layer 2: run-level metrics

Roll each run up to a record. This is the layer most teams skip, and it's what turns traces into a system view.

{ run_id, outcome: completed|escalated|halted|failed,
  turns, tool_calls, unique_tools_used,
  input_tokens, cached_tokens, output_tokens, cost,
  wall_ms, model_ms, tool_ms,
  retries, tool_errors, budget_exhausted: bool,
  user_followup: none|rephrased|abandoned|accepted }

With this you can ask the questions that matter: which runs are expensive, which escalate, which take too long, whether any of that is getting worse. From traces alone, each of those requires a scan.

The outcome field deserves care: escalated, halted, and budget-exhausted are distinct outcomes and collapsing them into "failed" destroys your ability to tell a working escalation path from a broken agent.

Layer 3: the aggregate view

Small set of numbers, watched over time:

• Completion rate, split by outcome type.
• Turns per completed task, p50 and p95. The leading indicator — it moves before cost and before complaints.
• Cost per completed task, p50 and p95.
• Tool call distribution. Which tools, in what proportion. Shifts here reveal behavior changes nothing else surfaces.
• Tool error rate by tool. Isolates a degrading dependency from a degrading agent.
• Escalation rate, and how often the human's decision differs from what the agent proposed.
• Cache hit rate. Should be high and stable; a drop is almost always a context-ordering regression.

💡 The three signals that are agent-specific

Standard APM won't suggest these and they're the most diagnostic things you can track.

Turns per task distribution, not the mean. A bimodal distribution — most runs at four turns, a cluster at twenty — means two different behaviors are hiding under one average. That cluster is where your cost and quality problems both live, and the mean makes it invisible.

Tool sequence n-grams. Count the common two- and three-tool sequences. A new frequent sequence appearing after a deploy is a behavior change; a loop showing up as A→B→A→B is an agent stuck. This catches things no per-call metric does.

Context size at each turn. Growth per turn should be roughly predictable. A jump means something returned far more than expected — usually the cause of both a cost spike and a quality drop, and visible here turns before it's visible anywhere else.

Connect it to user outcomes

The highest-value field is the cheapest: what the user did next. Rephrased the same request, abandoned, escalated to a human, or accepted the result.

This is a free quality label at production volume, and it beats any judge score because it reflects what actually happened rather than what a model thought of the output. Feed the rephrase and abandon cases into your eval set — they're your suite's best source of real failures.

✅ Alert on derivatives, not levels

• Turns per task up more than X% week over week.
• Cost per completed task up more than X%.
• Tool error rate for any single tool above its own baseline.
• Escalation rate moving sharply in either direction.
• Cache hit rate dropping.

Absolute thresholds go stale as traffic changes; rates of change stay meaningful. And an alert on any tool's own error rate catches a single degrading dependency that an aggregate would dilute into nothing.

The takeaway

A trace explains one run. Run-level records explain your system, and aggregates explain its direction. Capture the rendered arrays and full tool results at layer one, roll every run into a structured record at layer two, and watch turns, cost, tool distribution, and cache hit rate at layer three. Then record what the user did next — it's one field, it's free, and it's the best quality signal you'll get.