What a System Prompt Should and Shouldn't Contain
Five things belong in a system prompt. Most of what accumulates there — tool instructions, deterministic rules, incident patches, data — belongs somewhere it would actually work.
System prompts grow by accretion. Each incident adds a line, each edge case adds a paragraph, and after a few months there's a two-thousand-word document that nobody wants to touch because it's unclear which parts are load-bearing.
The useful discipline is knowing what belongs there at all. Most of what accumulates in a system prompt belongs somewhere else.
The five things that belong
1. Role and scope. What this agent is for, and — more usefully — what it isn't. Two or three sentences. The negative half does more work than the positive: "This agent handles billing questions. It does not process refunds or modify accounts" prevents a class of attempts that the positive statement alone doesn't.
2. Standing constraints. Rules that apply to every turn and can't be expressed in code. Not "validate the email address" (that's the tool's job) but genuine judgment constraints: "When a request could be interpreted two ways and one interpretation involves spending money, ask rather than assume."
3. Output requirements. Format, length, structure, what to include. Only where the model has genuine latitude — if the output is parsed, enforce the schema in code and mention it briefly here.
4. How to handle ambiguity and failure. What to do when information is missing, when a tool fails, when the request is out of scope. Agents behave much better with this stated explicitly, and it's the section most often absent. Without it, the default is confident guessing.
5. Domain context the model can't know. Your terminology, your entity relationships, the fact that "account" means something specific in your product. Short, factual, stable.
What doesn't belong
Tool usage instructions. "Use search_orders when the user asks about an order" belongs in the tool's description, where it's in context exactly when the tool is offered and stays with the tool when it changes.
Deterministic rules. "Always format dates as ISO 8601." If it must always happen, do it in code. A prompt line is advice; a formatter is a guarantee.
Long examples of the whole task. A few short examples of a specific tricky pattern help. Three full worked examples consume context on every turn and often teach the model to imitate their surface shape.
Data. Account details, retrieved documents, current state. These belong in the message array where they can be updated, dated, and removed. A system prompt with data in it is a system prompt that's stale.
Incident patches. "IMPORTANT: Never refund orders from before 2024 without checking the legacy system." This is a rule with a specific trigger — put it in the refund tool's description or, better, enforce it in the refund handler.
⚠️ That last category is the main source of bloat. Every one of those lines was added for a real reason, which is why nobody removes them. The fix is to relocate them, not delete them.
The accretion test
Once a system prompt passes a few hundred words, audit it line by line with three questions:
- Does this apply to every request? If it applies to a subset, it belongs with the tool, the route, or the code path for that subset.
- Is this deterministic? If yes, it belongs in code.
- Would removing it change behavior? Test it. A meaningful share of accumulated lines have no measurable effect — they were added alongside a real fix and got the credit.
That third one is the useful surprise. Run your eval suite with a line removed; if nothing changes, it was ceremony.
💡 Structure it for reading
The model reads it every turn, and so do you when debugging. A short prompt with clear sections is easier to attend to than a long one with nested caveats:
## Role
## Constraints (numbered, one rule per line)
## Output
## When you're unsure (the escalation and ambiguity rules)
## Domain notes
✅ Numbered constraints matter more than they look — they're referenceable in eval failures ("violated constraint 4") and in review discussions, which makes the whole thing maintainable.
Keep it stable
The system prompt is the first thing in the array, which makes it the prefix your cache depends on. Anything dynamic in it — a timestamp, a user name, the current date — invalidates caching for every call.
Put dynamic content in the message array instead. The system prompt should be byte-identical across every run of the same agent version.
The takeaway
A system prompt holds role, standing constraints, output requirements, ambiguity handling, and domain vocabulary. Tool instructions go with tools, deterministic rules go in code, data goes in the array, and incident patches go wherever the trigger lives. Audit by line with the three questions, and test whether removing something changes anything — a shorter prompt is usually a better-followed one.