#testing
Every post tagged "testing".
43 posts · showing 1–10
-
Your Research Agent Scored 100% on Citations. The Grader Only Checked That a Source Was Attached.
An eval that has never failed anything is not proof of a good agent. Before you trust a grader's score, feed it answers you know are wrong and see if it notices.
-
Your Research Agent Passed the Eval Once. Run Every Case Five Times and Grade the Worst Run.
A single green run tells you the agent can do the task, not that it will. Repeat each eval case, split the suite into solid, flaky and broken, and the flaky bucket becomes your real to-do list.
-
The Agent Got the Right Answer by the Wrong Route. Score the Trajectory, and Know the Three Cases Where the Answer Is Enough
A final-answer eval passes an agent that guessed, one that ran a write query mid-task, and one that took forty steps to do the job in four. Here is how to assert on the path without freezing a golden trace, and when you can safely skip it.
-
A Second Pass That Asks the Same Question Finds the Same Bugs. Change the Unit, Not the Temperature.
Raising the temperature on a review pass re-rolls the dice inside a frame that has already decided what is findable. Three properties determine what a pass can see, and a real second pass has to change at least two of them.
-
Your Eval Ran Each Case Once. That's Why It Missed the Flake.
A forty-case suite can pass every case and still hide a bug that reaches users one week in five. The number missing from most agent evals isn't coverage — it's how many times you ran each case.
-
Green Evals, Broken Production: The Frozen Fixture Problem
An eval suite built on recorded tool responses stops testing your agent and starts testing a museum. Here are three cheap detectors for fixture rot — and the rule for when a case should call the live tool instead.
-
Your 95% Step Accuracy Is a 60% Agent
Compounding explains part of the gap between a green eval dashboard and a broken agent. The rest is that per-step scoring counts a step as correct when it succeeds on a premise that was already wrong.
-
Test the Tool or Test the Agent? What Each Layer Catches
Sort a month of eval failures by root cause and most turn out to be tool contract bugs — findable in milliseconds by a unit test, currently being caught by the slowest instrument you own.
-
Reading the Agent Discourse Without Being Misled
Two questions handle most of it: what specifically was the input, and what specifically checked the output? Almost every overstated claim is weak on one of them.
-
Security Review When Nobody Wrote the Code
Human vulnerabilities carry a tell — hurried code, an obvious shortcut. Generated code is uniformly careful-looking, so scanning for the suspicious part stops working. But its consistency cuts both ways.