What "Agents Got Better" Would Actually Look Like in Your System

People wait for agents to improve without having decided what improvement would look like from where they're standing. That makes the waiting unfalsifiable — no arriving capability ever quite counts, and no roadmap ever gets unblocked.

Deciding in advance what would change is a small exercise that makes model releases actionable instead of atmospheric.

Write the conditions down first

For each thing your agent can't do, state the observable that would tell you it now can:

blocked: agent can't handle multi-intent support requests
would change if: it decomposes "cancel my order and update my address"
                 into two handled items rather than doing the first
                 and dropping the second
test:   12 multi-intent cases in the eval set; currently 3/12
threshold: 10/12 → route this class to the agent

That's falsifiable. Every model release, run the twelve cases. Either the number moved or it didn't — no debate about whether things feel better.

⚠️ Without a written threshold, evaluation of a new model becomes a vibe check on a handful of impressions, which is exactly how teams both over- and under-adopt.

What improvement tends to look like in practice

From the mechanism rather than from marketing, the changes that actually show up in a deployment:

Fewer turns for the same task. Higher per-step reliability means less retrying, less exploratory flailing. This is usually the first observable change, and it shows up as cost per completed task falling before quality visibly rises.

A different failure distribution, not just fewer failures. Failures concentrate more in genuinely hard cases and less in ordinary ones. Worth checking specifically — an aggregate score can improve while a category you care about regresses.

Scaffolding becoming unnecessary. Prompt workarounds for old limitations stop being load-bearing. ✅ This is real maintenance relief and it only materializes if someone goes back and tests whether the workaround is still needed.

Longer chains staying coherent. Tasks that fell apart at fifteen steps holding together at twenty-five.

Better behavior on the ambiguous middle. Asking rather than guessing, more often, if your prompt permits it.

What tends not to change

• Anything blocked on missing information. The deciding fact still isn't in any artifact.
• Anything blocked on absent verification. No oracle before, no oracle after.
• Anything blocked on permission. Trust moves socially, on its own schedule.
• Specification quality. Unchanged, and it's frequently the actual constraint.

→ Which is why "we'll revisit when models improve" is a reasonable plan for some blockers and a way of never revisiting for others.

🔍 The upgrade routine worth having

When a new model arrives:

1. Run the frozen eval set against old and new, same recorded tool responses. Compare pass rate, turns per task, and cost per task — all three.
2. Check the written thresholds. Did anything cross? Those are your unblocked items.
3. Read the diverging cases. Where the trajectory changed, understand why. Some will be improvements; some will reveal your tool descriptions were ambiguous all along.
4. Test the workarounds. Remove each one and see if behavior holds. Delete what's no longer needed.
5. Re-run the safety suite. Injection resistance and invariants don't automatically survive a version change.
6. Recalibrate thresholds that depend on model behavior — confidence routing especially.

An afternoon, and it converts a release from news into a decision.

💡 The uncomfortable finding

Teams that do this often discover the blocked list didn't move much — because most items were blocked on specification quality, missing context, or absent verification rather than on capability.

That's disappointing and useful. It redirects effort from waiting to building, and the things it points at — write better specs, capture the context, build the oracle — are things you can start this week and that compound with every future model improvement.

The takeaway

Decide in advance what improvement would look like: a written threshold on a real eval case, per blocked capability. Then each release is a measurement rather than an impression. Expect fewer turns before you see better answers, expect scaffolding to become removable, and expect anything blocked on missing information, missing verification, or missing permission to be exactly where you left it — which tells you where the work actually is.