#evaluation
Every post tagged "evaluation".
98 posts · showing 51–60
-
The Reliability Ceiling and What Would Have to Change
98% per step sounds excellent and produces 67% on a twenty-step task. The plateau is compounding arithmetic — which means the fix is shortening the chain, not improving the steps.
-
Predicting Agent Capability: Reason From Mechanism, Not Vibes
"Will agents get better?" is the wrong question. Six things block any given task, only two of them move with model releases — and the other four are the ones nobody's working on.
-
Novel Problems vs Recombined Ones: The Line That Matters
On genuinely novel problems you don't get an error — you get the nearest known pattern applied confidently to an adjacent problem. One classification decides where your attention belongs.
-
Ambiguity Resolution Stays a Human Job for a While
An agent doesn't fail on ambiguity — it absorbs it, picks a reading, and produces a confident answer to a question nobody asked. Better models guess better; they don't guess less.
-
Why Agents Struggle Where There's No Feedback Signal
Coding agents work well because compilers exist, not because code is easy. Effectiveness tracks whether a domain has a fast, independent, specific verdict — and that's frequently something you can build.
-
The Failure Modes That Don't Improve With Scale
Most agent limitations improve with the models. A smaller set won't, because the constraint isn't the model — and building elaborate workarounds for one while waiting out the other gets both wrong.
-
The SDLC Steps That Resist Automation Longest, and Why
Order the lifecycle by automatability and it doesn't track difficulty at all. Two properties decide it — and the steps that resist are short, frequent, and where all the leverage sits.
-
Code Review When Most Code Is Agent-Written
Review practices assumed an author who had reasons, understood the system, and made idiosyncratic mistakes. Agent output breaks all three — and the bugs that survive are the ones that look correct.
-
Agents Are Better at Writing Code Than at Deciding What to Build
Both feel like thinking about the problem, and only one has complete inputs and a checkable answer. The reframe that makes a model genuinely useful near a decision it shouldn't be making.
-
What "End-to-End Development Agent" Actually Means Right Now
The phrase covers demos that differ by orders of magnitude. Naming the eight stages shows which two are genuinely autonomous — and why chaining them is qualitatively harder than any one of them.