#evaluation
Every post tagged "evaluation".
35 posts · showing 11–20
-
Getting an Agent to Say How Sure It Is
Ask a model how confident it is and you get a plausible number that doesn't move with difficulty. Five signals that actually correlate with being right — and the calibration check that tells you if yours is noise.
-
Few-Shot Examples: When They Help an Agent and When They Hurt
Two or three similar examples is the worst case — enough to teach a template, not enough to convey the range. And the example nobody includes is the one where the right answer is to decline.
-
Local Models for Agent Work: Where They Actually Fit
"Can a local model replace the frontier one?" is the wrong question. Which steps can move is the right one — and the comparison people skip is against a small hosted model, where most of the saving already lives.
-
Bisecting a Bad Run: Finding the Change That Broke It
It worked last week. Six things could have changed, and they vary independently — so bisect across the dimensions first, then within one. And diff the raw arrays for what's missing, not what's different.
-
Treating Prompts Like Code: Versioning, Review, and Rollback
A one-line edit in a web console changes behavior across every request, with no history and no way back. Prompts in the repo, rendered hashes in the logs, eval results in the PR — and a quarterly prune.
-
The Migration Agent: 4,000 Files and What Made It Tractable
The difference between a migration agent that saves weeks and one that produces four thousand unreviewable diffs is decided before it touches a file — and it starts with building the checker, not the agent.
-
Testing Multi-Agent Systems Without Losing Your Mind
Add a second agent and end-to-end assertions stop telling you anything useful. Contracts at every seam, stubs for the supervisor's unhappy paths, and integration tests that assert on graph shape and cost.
-
The Verifier in the Loop: Designing an Oracle Your Agent Can Iterate Against
Coding agents work as well as they do because compilers exist. In other domains the oracle is something you build — and most tasks have one hiding in them if you spend ten minutes looking.
-
Where Agents Don't Belong (Yet)
Six task shapes where the agent architecture is the wrong fit regardless of model quality — each with a tell you can check in an afternoon, before committing a quarter to it.
-
The Code-Review Agent That Commented on Everything
Nine comments on a twelve-line PR and the team muted it. Any agent asked to find things will find them — the fix is a hard cap that forces ranking, and pointing it at what humans actually miss.