Why Agents Struggle Where There's No Feedback Signal

Coding agents are unusually effective, and the usual explanation is that models are good at code. The better explanation is that code comes with an oracle attached. It compiles or it doesn't. Tests pass or they don't. The agent gets a definitive, immediate, free verdict on every attempt, and can iterate against it.

Remove that verdict and performance drops sharply, in ways that have nothing to do with the difficulty of the domain.

What the oracle actually provides

Three things, all necessary:

Ground truth independent of the model's own judgment. The compiler doesn't care how confident the agent is. That independence is what makes the signal informative rather than self-confirming.

Immediacy. Feedback within seconds means many iterations. A signal arriving in weeks means one attempt.

Specificity. "Type error at line 40: expected Decimal, got float" localizes the problem. "This is wrong" doesn't, and the agent's next attempt is a guess.

Take any domain, ask whether it has all three, and you can predict agent effectiveness there better than by asking how hard the domain is.

The domains sorted by oracle quality

Strong oracle — code (compilers, tests), structured extraction (schema validation, quoted-span checking), data reconciliation (totals must match), configuration (validators), SQL (the query runs or doesn't).

Weak oracle — writing (readable is not checkable), design (multiple valid answers), summarization (faithful is checkable, useful isn't), classification with fuzzy boundaries.

No oracle — prioritization, architecture decisions whose rightness depends on the future, strategy, hiring, anything evaluated over months and confounded by everything else that happened.

⚠️ Notice this ordering doesn't track intellectual difficulty. Writing a correct concurrent data structure is harder than deciding which feature to build next, and agents do far better at the first.

What happens without one

The agent produces its best attempt and stops, because there's nothing to iterate against. That single attempt is often good — but it's a sample, not a converged answer, and it arrives with the same confident tone a verified answer would have.

Worse, self-evaluation substitutes silently. Asked to check its own work with no external signal, an agent produces an assessment that shares the blind spots that produced the work. Agreement between the attempt and the self-check is not evidence.

✅ Building the oracle is the highest-leverage move

The productive response isn't to avoid oracle-free domains. It's to notice that oracles are frequently constructible, and that constructing one converts a weak domain into a strong one.

• Schema validation for any structured output. Free, instant, specific.
• Quoted-span verification — every claim must include a verbatim span from a source, checked as a literal substring. Converts summarization and research from weak-oracle to strong.
• Arithmetic reconciliation — totals, balances, counts that must line up between input and output.
• Round-trip identity — transform, invert, compare. Catches lossy transformations.
• Constraint satisfaction — no overlaps, within budget, dependencies respected.
• Cross-source agreement — the extracted value must match the system of record.

→ Ten minutes spent looking for a constructible oracle in a domain that seems to lack one is usually better spent than a day of prompt engineering, because it changes the shape of the problem rather than nudging the output.

When there genuinely isn't one

Some things can't be made checkable, and that's important information rather than a failure:

• Use the agent to enumerate, not to decide. Options and trade-offs play to recall and structuring; the choice goes to whoever holds the context.
• Require assumptions to be stated, so the unsupported parts of an answer are visible.
• Sample for agreement across independent attempts — weaker than an oracle, and it's a real signal about whether the question is hard.
• Keep a human in the loop, and accept that this is the correct answer rather than a temporary one.

🔍 The diagnostic

For any task you're considering handing to an agent: what tells it that it got the answer wrong, and how fast?

• Seconds, definitive, specific → expect good results.
• Slow or fuzzy → expect a plausible single attempt.
• Nothing → expect a confident guess, and design accordingly.

The takeaway

Agent effectiveness tracks oracle availability more than domain difficulty. Code is the flagship case because compilers and tests are an unusually good oracle, not because code is easy. Before improving a prompt, ask what tells the agent it's wrong — and if the answer is nothing, spend the effort building the check instead. That's the move that changes what's possible rather than what's likely.