Testing & Reliability
Evals, debugging, tracing, why agents fail.
30 posts · showing 11–20
-
Your 95% Step Accuracy Is a 60% Agent
Compounding explains part of the gap between a green eval dashboard and a broken agent. The rest is that per-step scoring counts a step as correct when it succeeds on a premise that was already wrong.
-
Test the Tool or Test the Agent? What Each Layer Catches
Sort a month of eval failures by root cause and most turn out to be tool contract bugs — findable in milliseconds by a unit test, currently being caught by the slowest instrument you own.
-
When to Stop Improving an Agent
There's always another failing case, so without a stopping rule the work either never ends or ends arbitrarily. The signal isn't the error rate — it's effort per point of improvement.
-
What Happens to QA When Agents Write the Tests
Cheap test generation makes suite quality matter more, not less — because the default outcome is a large, green suite that verifies nothing. Someone has to own whether it catches things.
-
Evaluation Engineer: The Job Description Writing Itself
The furthest-formed of the emerging roles: well-defined, consistently needed, consistently unstaffed. It's everyone's dependency and nobody's job, which is exactly why.
-
How to Benchmark Models on Your Own Task
Public benchmarks tell you about a model, not about your system. Frozen tool responses, assertions written from the requirement, and four numbers instead of one — plus the shortcut when a full suite isn't happening.
-
What Happens When Verification Gets Automated Too
"Agents generate, humans verify" is stable partly because verification is hard to automate. Parts of it aren't — and the part that survives isn't reviewing, it's knowing what was wanted.
-
The Reliability Ceiling and What Would Have to Change
98% per step sounds excellent and produces 67% on a twenty-step task. The plateau is compounding arithmetic — which means the fix is shortening the chain, not improving the steps.
-
Debugging Code You Didn't Write and Don't Understand
The instinct is to hand the failure back to the agent. Without a reproduction that produces a plausible fix to a plausible cause — and two rounds later nobody understands the module at all.
-
Code Review When Most Code Is Agent-Written
Review practices assumed an author who had reasons, understood the system, and made idiosyncratic mistakes. Agent output breaks all three — and the bugs that survive are the ones that look correct.