#testing
Every post tagged "testing".
43 posts · showing 11–20
-
What Happens to QA When Agents Write the Tests
Cheap test generation makes suite quality matter more, not less — because the default outcome is a large, green suite that verifies nothing. Someone has to own whether it catches things.
-
Evaluation Engineer: The Job Description Writing Itself
The furthest-formed of the emerging roles: well-defined, consistently needed, consistently unstaffed. It's everyone's dependency and nobody's job, which is exactly why.
-
Model, Loop, Harness: Three Layers and Who Owns Each
Three layers, three change rates, three skill requirements. Conflating the loop and the harness is why loop bugs get fixed with prompt edits — and why domain experts can't fix domain problems.
-
How to Benchmark Models on Your Own Task
Public benchmarks tell you about a model, not about your system. Frozen tool responses, assertions written from the requirement, and four numbers instead of one — plus the shortcut when a full suite isn't happening.
-
When a Small Model Beats a Big One, and How to Tell
Sometimes "beats" means cheaper at acceptable quality — and sometimes it means genuinely better on the metric that matters. Four cases where smaller wins outright, and four where it looks like it should and doesn't.
-
What Happens When Verification Gets Automated Too
"Agents generate, humans verify" is stable partly because verification is hard to automate. Parts of it aren't — and the part that survives isn't reviewing, it's knowing what was wanted.
-
What "Agents Got Better" Would Actually Look Like in Your System
Waiting for agents to improve without deciding what improvement looks like makes the waiting unfalsifiable. Write the threshold down first, and every release becomes a measurement instead of a vibe.
-
The Reliability Ceiling and What Would Have to Change
98% per step sounds excellent and produces 67% on a twenty-step task. The plateau is compounding arithmetic — which means the fix is shortening the chain, not improving the steps.
-
Debugging Code You Didn't Write and Don't Understand
The instinct is to hand the failure back to the agent. Without a reproduction that produces a plausible fix to a plausible cause — and two rounds later nobody understands the module at all.
-
When the Requirement Itself Is Wrong, Who Notices?
The behavior was intentional, decided two years ago, documented somewhere nobody looked. Every stage faithfully executed a wrong ticket — and every automated check confirmed it, by construction.