Predicting Agent Capability: Reason From Mechanism, Not Vibes

Predictions about what agents will do in two years mostly take one of two forms: extrapolate a trend line, or extrapolate a feeling. Both produce confident numbers and neither survives contact with the specific question you actually have, which is whether this task will be automatable in your system.

A more useful method is to reason about what each capability requires and ask which of those requirements is moving.

The method

For any task an agent does badly, identify what's actually blocking it. There are only a handful of candidates, and they move at very different rates.

1. Raw model capability — reasoning quality, instruction adherence, tool selection. Moving fast, improving on a schedule you don't control. If this is the blocker, waiting is a reasonable strategy.

2. Effective context handling — using information that's present rather than just fitting it. Improving, more slowly than context size, and the gap between the two is where a lot of disappointment lives.

3. Information availability — whether the deciding facts exist in any readable artifact. Moves only as fast as your organization records things. Largely under your control, and largely not being worked on.

4. Feedback signal — whether correctness is checkable. Doesn't move with models at all. Moves when someone builds an oracle, which is engineering work you can do this week.

5. Trust and permission — whether anyone will allow autonomous action. Moves with demonstrated reliability and legible failure modes, on a social timescale.

6. Accountability structure — whether a decision can be made by a non-party. Moves with law and organizational norms. Slowest of all.

→ The useful question isn't "will agents get better?" It's which of these six is my blocker, because the answer determines whether you wait, build, or restructure.

Applying it

"Agents can't handle our deployment decisions." Blocker is 3 (system state, calendar, who's available — mostly unrecorded) and 6 (someone must own it). Neither moves with model capability. Waiting is the wrong strategy; building better readiness reporting is the right one.

"Agents write tests that don't catch anything." Blocker is 4 — no feedback on whether a test is meaningful. Constructible: mutate the implementation and check the test fails. Build it and this moves from blocked to solved, independent of models.

"Agents pick the wrong tool from our fifty." Blocker is 1, partly, and mostly a design problem — fifty tools is a surface issue. Fix the surface; expect the rest to improve on its own.

"Agents lose the thread on long tasks." Blocker is 2, plus context assembly. Improving, and a scratchpad plus structured state addresses most of it now.

"Nobody will let the agent touch production." Blocker is 5. Moves through demonstrated reliability, bounded blast radius, and reversibility — none of which are model improvements.

⚠️ Two failure modes in prediction

Attributing a structural blocker to capability. "Once models are better, it'll know what to prioritize." Prioritization is blocked on 3, 4, and 6 simultaneously. That sentence describes something arriving from a direction it isn't coming from, and the time spent waiting is time not spent building the oracle or capturing the context.

Attributing a capability blocker to structure. Building elaborate scaffolding for something that improves next quarter leaves you maintaining machinery for a solved problem.

The test is the same in both directions: if the model were substantially better, would this work?

💡 What the method says about timing

The blockers you control — information availability and feedback signal — are the ones nobody is working on, and they're the ones that determine whether model improvements land in your system at all.

A better model in an environment with no oracle and no recorded context produces better-argued guesses. The same model with a good oracle and well-captured context produces verified work. → The capability improvements are free and arriving; the environment improvements are yours and aren't.

Which suggests where to spend: not on prompt tuning for current limitations, but on building the checks and capturing the context, so that each model improvement translates into something.

✅ The prediction worth making

Not a date. A conditional: this task becomes automatable when [specific blocker] is addressed, and that blocker moves via [specific mechanism].

That form is falsifiable, actionable, and tells you whether you're waiting or building.

The takeaway

Trend extrapolation and vibes both produce confident predictions about the wrong question. Identify which of six blockers applies — capability, effective context, information availability, feedback signal, trust, accountability — and note that only the first two move with models. The rest move because you move them, and they're what determines whether the improvements you're waiting for will actually help you when they land.