#evaluation
Every post tagged "evaluation".
35 posts · showing 21–30
-
Canary Releases for Prompt Changes
A sentence added to fix one behavior routinely changes three others. Ship prompts like behavior: versioned by rendered hash, split by stable key, and compared on tool distribution — the row nobody collects.
-
Chunking for Agents Is Not Chunking for Search
Search results are read by a person who fills in the gaps. An agent doesn't — it acts on the fragment. That one difference changes how you chunk, how many you return, and what "no results" should mean.
-
Migrating Model Versions Without Breaking Your Agent
The new model is better at nearly everything and different at everything, and tool selection is what shifts first. Replay a frozen eval set against both — and put the cost-per-task row in the comparison.
-
Model Routing: Using the Cheap Model for the Cheap Steps
Classification, extraction, and formatting move to a cheaper tier safely. Tool selection and planning don't — and the failure looks like a tool-description bug, so you'll debug the wrong thing for a week.
-
Building an Eval Set From Production Traces
Hand-written cases test what you thought of. A user rephrasing the same request three times is a labelled failure sitting in your logs — here's how to turn a week of traces into a suite that catches the real ones.
-
The Demo-to-Production Gap Is an Evaluation Gap
Demos answer "can it?" Deployments ask "how often, across everything real users send, when the tools misbehave?" Five variants of your demo scenario will tell you more than the demo did.
-
Flaky Agents: Separating Model Nondeterminism From Your Bugs
A test passed on re-run. That's ambiguous for agents in a way it isn't for other software — and the fix is to stop reporting pass/fail and start reporting distributions.
-
Debate or Vote? Getting Real Value From Multiple Model Opinions
Voting harvests independent error; debate harvests challenge. Pick wrong and you multiply your token bill for agreement you'd have had from a single call — and the disagreement, the part worth paying for, gets logged away.
-
One Prompt Per Job Beats One Prompt For All: The Router Pattern
Every new case adds a paragraph, and adherence to all of them drops. Classify first and run a small prompt built for that class — with per-route tools, limits, and approval policy.
-
Build or Buy the Eval Harness? What Each Choice Locks In
Teams buy an eval platform and keep their own scripts anyway. The reason is deterministic replay — it hooks into your dispatch layer, which is the one part no vendor can own.