#evaluation
Every post tagged "evaluation".
98 posts · showing 81–90
-
The Verifier in the Loop: Designing an Oracle Your Agent Can Iterate Against
Coding agents work as well as they do because compilers exist. In other domains the oracle is something you build — and most tasks have one hiding in them if you spend ten minutes looking.
-
Where Agents Don't Belong (Yet)
Six task shapes where the agent architecture is the wrong fit regardless of model quality — each with a tell you can check in an afternoon, before committing a quarter to it.
-
The Code-Review Agent That Commented on Everything
Nine comments on a twelve-line PR and the team muted it. Any agent asked to find things will find them — the fix is a hard cap that forces ranking, and pointing it at what humans actually miss.
-
Canary Releases for Prompt Changes
A sentence added to fix one behavior routinely changes three others. Ship prompts like behavior: versioned by rendered hash, split by stable key, and compared on tool distribution — the row nobody collects.
-
Chunking for Agents Is Not Chunking for Search
Search results are read by a person who fills in the gaps. An agent doesn't — it acts on the fragment. That one difference changes how you chunk, how many you return, and what "no results" should mean.
-
Migrating Model Versions Without Breaking Your Agent
The new model is better at nearly everything and different at everything, and tool selection is what shifts first. Replay a frozen eval set against both — and put the cost-per-task row in the comparison.
-
Model Routing: Using the Cheap Model for the Cheap Steps
Classification, extraction, and formatting move to a cheaper tier safely. Tool selection and planning don't — and the failure looks like a tool-description bug, so you'll debug the wrong thing for a week.
-
Building an Eval Set From Production Traces
Hand-written cases test what you thought of. A user rephrasing the same request three times is a labelled failure sitting in your logs — here's how to turn a week of traces into a suite that catches the real ones.
-
The Demo-to-Production Gap Is an Evaluation Gap
Demos answer "can it?" Deployments ask "how often, across everything real users send, when the tools misbehave?" Five variants of your demo scenario will tell you more than the demo did.
-
Flaky Agents: Separating Model Nondeterminism From Your Bugs
A test passed on re-run. That's ambiguous for agents in a way it isn't for other software — and the fix is to stop reporting pass/fail and start reporting distributions.