Reading the Agent Discourse Without Being Misled

Claims about agent capability arrive constantly and vary enormously in how much they support. A few questions separate the ones worth acting on from the ones worth noting, and they're the same questions regardless of who's making the claim.

For a demo

How good was the input? A demo on a well-specified task is showing you the easy half. Ask what the ticket or prompt actually said — the quality of the input frequently is the result.

Was the codebase or corpus well-patterned? Agents are far stronger where an existing pattern can be followed. A demo on a clean example repository doesn't transfer to a system with fifteen years of history.

What's the oracle? If nothing verified the output mechanically, "it produced something" isn't "it produced something correct."

How many attempts? A demo is usually the successful run. The interesting number is how many were discarded.

What happens on the cases it can't do? Silent wrong output or a visible stop? A demo never shows this and it determines whether the thing is operable.

For a benchmark

Does the benchmark resemble your task? Benchmarks test well-posed, self-contained, checkable problems. If your problem is ambiguous or context-dependent, the number is about a different activity.

What's the variance? A single-run score on a stochastic system is one sample.

What's the cost and turn count? A version scoring higher and costing substantially more per task is a trade-off, not an upgrade. This row is usually missing.

Is the comparison paired? Independent pass rates on small suites can't distinguish small differences.

For a case study

⚠️ Who is telling you, and what do they sell? Not disqualifying, and it should recalibrate how much specificity you require.

Are there numbers, and are they per completed task? "3× faster" measured on implementation while delivery time didn't move is a real and misleading figure.

What's the denominator? Faster than what baseline, measured how?

What happened to review, integration, and maintenance? Studies that measure the compressed phase and not the expanded ones are measuring half the system.

How long did it run? Enthusiasm is highest in the first month. A result at three months means more.

For a strong claim about the future

Is the reasoning from mechanism or from trend? "Capability has improved, therefore this will be possible" is extrapolation. "This is blocked on X, and X moves via Y" is reasoning you can evaluate.

What would falsify it? A prediction with no disconfirming evidence isn't a prediction.

Does it distinguish capability blockers from structural ones? Claims that agents will soon handle prioritization or judgment usually don't — they assume a feedback signal that doesn't exist.

✅ The general filter

Two questions cover most of it:

What specifically was the input, and what specifically checked the output?

Almost every overstated claim is weak on one of those. The demo had a hand-written specification; the study had no verification; the benchmark tested something well-posed. Neither question is hostile, and both are answerable by anyone whose claim is solid.

💡 The claims worth taking seriously

Not the impressive ones — the specific ones:

• Bounded scope, stated clearly.
• A named oracle.
• Numbers per completed task, including cost.
• An account of what it doesn't do.
• Failure modes described.

A claim that includes its own limitations is usually from someone who ran the thing in production, and that's the signal worth weighting.

The takeaway

Ask what the input was and what checked the output — most overstated claims are weak on one of the two. For demos, ask about specification quality and the discarded attempts. For benchmarks, ask whether the task resembles yours and what the cost was. For case studies, ask what happened to review and maintenance. And weight claims that describe their own limits, because those come from people who operated the thing rather than demonstrated it.