Getting an Agent to Say How Sure It Is

A calibrated confidence signal is the single most useful thing an agent can produce beyond its answer. It's what lets you route the uncertain cases to a human, escalate selectively, and let the agent proceed autonomously on the rest.

Asking a model "how confident are you?" mostly doesn't produce it. Self-reported confidence clusters high, moves little, and correlates weakly with being right. That doesn't mean the signal is unavailable — it means the useful ones come from elsewhere.

Why the direct question fails

A model asked to rate its confidence produces a plausible-sounding number in the same way it produces everything else. Nothing in that process consults an internal error estimate, because there isn't one to consult in the way the question implies.

The characteristic result: nearly everything reported as high confidence, including the wrong answers, and the same distribution regardless of how hard the case actually was. A number that doesn't vary with difficulty can't be used to route by difficulty.

Five signals that work better

1. Agreement across independent samples. Run the same question several times with varied framing. Unanimity is meaningful evidence; a split is a genuinely hard case. This is the most reliable confidence signal available, and it costs N times a single call — worth it for consequential decisions, not for everything.

2. Whether the evidence exists. Instead of asking how sure it is, ask what it's relying on:

{ answer: ...,
  supporting_facts: [{claim, source, quoted_span}],
  assumptions_made: [...],
  what_would_change_this: "..." }

An answer with three quoted sources and no assumptions is better-supported than one with an empty supporting_facts array and two assumptions. Both are checkable in code — no trust in self-assessment required. → The assumptions_made field alone is a strong triage signal.

3. Structural properties of the run. The trace carries information about difficulty: unusually many turns, a tool that failed and was worked around, a retrieval that returned nothing, the agent revising its plan twice. A run that struggled is a run to review, and every one of these is measurable from the trace without asking the model anything.

4. Verifier results. Where an oracle exists, "passed all checks" versus "passed after three attempts" versus "escalated" is a real confidence gradient grounded in something objective.

5. Distance from what it's seen. Retrieval scores below a threshold, an input unlike anything in the eval set, an entity the memory store has never encountered. Out-of-distribution detection, doing the same job it does elsewhere.

Make it a discrete routing decision

Even a good signal is more useful as a small number of buckets than a number. Ask the agent to classify rather than to score:

confidence: enum[
  "verified",      # checked against a source; can quote it
  "inferred",      # reasoned from available evidence, not directly stated
  "assumed",       # filled a gap; the assumption is listed
  "unknown"        # could not determine
]

Categories with definitions get applied far more consistently than a 0-100 scale, and they map directly onto what you'll do: verified proceeds, assumed gets flagged in the output, unknown escalates. ✅ Requiring per-claim confidence rather than a whole-answer score is better still — real answers are mixtures, and a single number averages away the one uncertain part that matters.

Calibrate it, or don't use it

Whatever signal you choose, check it against outcomes: group by reported confidence and measure actual accuracy in each group.

verified:  96% correct  (n=210)
inferred:  81% correct  (n=145)
assumed:   54% correct  (n=62)    ← behaving as intended
unknown:   —            (escalated)

That's a usable signal. If instead every bucket sits near the same accuracy, the signal is noise regardless of how sensible the categories look, and routing on it does nothing but add latency.

⚠️ Recalibrate after any model or prompt change. Confidence behavior shifts with both, and a threshold tuned six months ago may be routing the wrong things now.

The takeaway

Don't ask for a confidence score; build one. Sample agreement for consequential decisions, require quoted evidence and explicit assumptions, read difficulty signals out of the trace, and use verifier results where an oracle exists. Express it as a few defined categories rather than a number, attach it per claim, and check calibration against real outcomes — an uncalibrated confidence signal is worse than none, because it gets trusted.