What I'd Tell a Team Starting Their First Agent Project

The mistakes teams make on a first agent project are consistent enough to list, and most of them cost a quarter before they're visible. None is about model capability.

Pick a task where you can check the answer

Before anything else, ask how you'd know whether the output is right, and whether you could check a hundred of them in an afternoon.

If the answer requires an expert reviewing each case slowly, you cannot evaluate the system, which means you cannot improve it or notice when it degrades. You'll be running on impressions.

This constraint eliminates more candidate projects than any other, and it should be applied at the idea stage rather than after two months of building. → Tasks with a natural oracle — something that compiles, validates, reconciles, or matches a source of record — are worth a great deal more than they first appear.

Build the eval set before the agent

Not after the demo works. Before.

A hundred real inputs, drawn from actual traffic if any exists, including the ugly ones: ambiguous requests, multi-intent requests, cases where the right answer is to ask a question, cases where a tool fails. That set is your specification, and writing it clarifies what you're building more than any design document.

Teams that skip this spend the project tuning against impressions, and discover the real distribution at launch.

Start smaller than feels worthwhile

Three read-only tools, a loop you wrote, one task. Not because ambition is bad, but because you need to see the failure modes on something you fully understand before you can recognize them in something you don't.

The complicated version built first half-works, and nobody can tell which part is wrong.

Expect the work to be in tools and context, not prompts

The instinctive model is that agent quality comes from prompt engineering. In practice most of it comes from:

Tool descriptions that make selection unambiguous.
Tool results formatted for a model — explicit units, honest empties, actionable errors.
Context assembly that keeps the right things present at the right position.
Deterministic logic in code rather than as instructions.

⚠️ If your response to every failure is editing the prompt, you'll build a two-thousand-word document that nobody dares change and that fixes progressively less.

Decide the blast radius on day one

Read-only until you have evidence, then narrow write access with policy checks in the handlers. Not because agents are dangerous in the abstract, but because being able to say "the worst case is a wrong answer" is what lets you deploy quickly and iterate — and what lets others trust it.

The teams that move fastest are usually the ones who bounded the downside first.

Instrument before you need it

Cheap now, impossible retroactively:

The full rendered array on every call.
Every tool call with arguments and complete results.
Run-level records: outcome, turns, cost, tool sequence.
What the user did next — accepted, rephrased, abandoned.

That last one is a free quality label at production volume, and it's the best source of eval cases you'll ever have.

Plan for the failures that are quiet

The loud failures — errors, timeouts — get handled because they're visible. The ones that hurt are silent:

Partial work reported as complete.
An empty tool result read as a negative result.
A constraint dropped from context around turn twenty.
A confident answer where the correct response was a question.

Each needs a specific mechanism: explicit completion checking, tool results that distinguish empty from failed, standing constraints held outside the compactable history, and eval cases where declining is right.

✅ The short version

Choose a task whose output you can check cheaply.
Write the eval set first, from real inputs, including the ugly ones.
Build the smallest version, with a loop you understand.
Put the work into tools, results, and context — not into prompt length.
Start read-only; earn write access.
Log the rendered arrays, the run records, and what the user did next.
Build specific mechanisms for the silent failures.

The takeaway

None of this is about models, and it will stay true as they improve. The differences between agent projects that work and ones that stall are almost entirely about whether the team can tell what their system is doing — which comes from an eval set built early, instrumentation added before it's needed, and a scope small enough to understand. Get those right and the rest is ordinary engineering.