Few-Shot Examples: When They Help an Agent and When They Hurt

Adding examples to a prompt is the standard response to an agent behaving wrongly, and it often works. It also has a specific failure mode that's hard to spot afterwards: the agent starts treating the examples as the boundary of what's possible, and handles anything unlike them worse than it did before.

What examples actually teach

An example demonstrates a mapping from a situation to a response. The model generalizes from it — and what it generalizes is not necessarily what you intended.

Show three examples where the agent looks up an order before answering, and you may teach "check the order first." You may equally teach "questions are about orders," or "always call exactly one tool before answering," or something about the phrasing of your example inputs.

The narrower and more consistent your examples, the stronger and less controllable this pull becomes.

Where they clearly help

Output format, when it's unusual. One example of an odd structure teaches faster than a paragraph of description.

Edge-case handling. A single example of the tricky case — an ambiguous request where the correct response is a clarifying question — is worth more than any amount of instruction, because it demonstrates that the response is available.

Domain conventions. How your organization phrases things, what level of detail is expected, what your terminology means in practice.

Tone, which is nearly impossible to specify and trivial to demonstrate.

Where they hurt

When they narrow the perceived task. Three examples about billing, and the agent handles a shipping question as though it were about billing. → This is the main failure, and it's invisible unless your eval set contains cases unlike your examples.

When they're too similar to each other. Similar examples teach a template. The agent then produces that template even when the situation calls for something else, and the output looks confident and formulaic.

When they're long. Full worked examples are re-sent on every turn, cost real context, and — being long — exert a strong pull on the response shape.

When they're stale. An example demonstrating a workflow that changed six months ago teaches the old workflow, and it will outvote a corrected instruction elsewhere in the prompt. ⚠️ Examples drift out of date silently, because nobody reviews them when the process changes.

Practical rules

Prefer many-and-varied or none. Two or three similar examples is the worst case — enough to create a template, not enough to convey the range. If you use examples, deliberately vary the situation, the shape of the response, and the difficulty.

Include a negative example. Show a case where the correct answer is to decline, ask, or escalate. Without one, examples uniformly demonstrate confident action and the agent infers that's always right.

Keep them short. An example that shows the key move — a tool call with its arguments, a two-line response — teaches nearly as much as a full transcript at a fraction of the cost.

Put them where they apply. Examples of using a tool belong in that tool's description, not in the system prompt. Then they're present exactly when relevant, and they don't pull on unrelated requests.

Date them and review them. Add a comment with when and why each was added. Delete ones whose reason has expired.

🔍 The test

Take an example out and run your eval suite.

Most examples in mature prompts have no measurable effect — they were added alongside a real fix and inherited the credit. Removing them shortens the prompt, weakens the template pull, and costs nothing.

Then the complementary test: add an eval case deliberately unlike your examples. If performance on it is notably worse than on similar cases, the examples are narrowing the task, and that's a stronger reason to trim than any cost argument.

✅ On harvesting examples from production

Tempting and useful — real examples beat invented ones. Two cautions:

• Never across tenants. One customer's data becoming another's prompt is a leak, however anonymized it feels.
• Verify each one is actually correct. A production run that looked fine isn't necessarily a good demonstration, and an example encoding a subtle mistake teaches that mistake to every future run.

The takeaway

Examples teach a mapping, and the model decides what to generalize. Use them for format, edge cases, and tone; keep them short, varied, and placed with the tool or route they apply to; always include one where the right answer is to decline or ask. Then test by removing them — the ones that change nothing are costing you context and narrowing your agent for free.