#evaluation
Every post tagged "evaluation".
98 posts · showing 41–50
-
How to Benchmark Models on Your Own Task
Public benchmarks tell you about a model, not about your system. Frozen tool responses, assertions written from the requirement, and four numbers instead of one — plus the shortcut when a full suite isn't happening.
-
When a Small Model Beats a Big One, and How to Tell
Sometimes "beats" means cheaper at acceptable quality — and sometimes it means genuinely better on the metric that matters. Four cases where smaller wins outright, and four where it looks like it should and doesn't.
-
The Cost of Choosing Your Model Too Early
The choice is rarely the mistake — the accumulated compensations are. Prompts absorb quirks, the harness papers over weaknesses, and none of it looks like coupling until you try to change something.
-
Matching Model Tier to Task: A Practical Table
"Simpler tasks get cheaper models" isn't actionable, because simplicity isn't the predictor. Two properties are — and one row of the table overrides all the others.
-
Should You Always Reach for the Flagship Model?
"Always use the cheapest model that works" has costs that don't appear in a token bill. The case for flagship-by-default is stronger than usually admitted — and there's a cheaper optimization people skip first.
-
Three Things That Would Genuinely Surprise Me in Two Years
More useful than predictions: naming what would count as a genuine surprise. Three of them, each identifying a load-bearing belief — and what evidence would overturn it.
-
The Compounding Loop: Agents Building Agent Infrastructure
Agents are good at building the harness that makes agents useful — well-specified, verifiable plumbing. The loop is real, and it has a hard boundary at everything requiring judgment about what to build.
-
What Happens When Verification Gets Automated Too
"Agents generate, humans verify" is stable partly because verification is hard to automate. Parts of it aren't — and the part that survives isn't reviewing, it's knowing what was wanted.
-
What "Agents Got Better" Would Actually Look Like in Your System
Waiting for agents to improve without deciding what improvement looks like makes the waiting unfalsifiable. Write the threshold down first, and every release becomes a measurement instead of a vibe.
-
Why Benchmarks Keep Rising While Deployments Stay Hard
Both reports are accurate. Benchmarks test well-posed, self-contained, checkable problems — and almost none of deployment difficulty is any of those things.