Model Routing: Using the Cheap Model for the Cheap Steps
Classification, extraction, and formatting move to a cheaper tier safely. Tool selection and planning don't — and the failure looks like a tool-description bug, so you'll debug the wrong thing for a week.
An agent run is not one workload. Classifying an intent, extracting fields from a document, deciding which tool to call next, and synthesizing a final answer are different problems with different difficulty, and running all of them on your most capable model is the simplest available way to overspend.
Routing steps to different models is straightforward. Doing it without quietly degrading quality requires knowing which steps are safe to move.
Which steps move well
Classification and routing. Small, closed label sets. A capable small model matches a large one on well-specified classification, and this is often the highest-frequency call in the system.
Structured extraction. Pulling fields from a document into a schema. Constrained by the schema, verifiable in code, and cheap to check — if the extract validates and the quoted spans match the source, it's right regardless of which model produced it.
Formatting and mechanical transformation. Rewriting into a template, converting formats, applying a fixed style. No judgment involved.
Summarizing a single document. Usually fine on a smaller model, particularly when the summary feeds a further step rather than a person.
Which steps don't
Tool selection in a wide tool space. This is where capability differences show most sharply. A weaker model picks a plausible neighbouring tool, and the resulting failure is expensive and hard to attribute — it looks like a tool-description problem, so you'll spend a week rewriting descriptions.
Multi-step planning under ambiguity. Deciding what to do next, when the answer isn't obvious, is exactly the capability you're paying for.
Final synthesis, when a person reads it. Quality differences that don't show in intermediate steps show here.
Anything with a destructive side effect. Not because a small model can't do it, but because the cost asymmetry is wrong. Save money on the twenty reads; don't save it on the one write.
Implementing without a mess
Route by step, declared in configuration, not by heuristics computed at runtime:
STEP_MODELS = {
"classify_intent": SMALL,
"extract_fields": SMALL,
"select_tool": LARGE,
"plan": LARGE,
"summarize_doc": SMALL,
"final_synthesis": LARGE,
}
Readable, testable, and changeable in one place. ⚠️ Avoid dynamic escalation-on-confidence as a first implementation — it sounds appealing and it means paying for the small model and the large one on every hard case, plus a confidence signal that's itself unreliable. Static routing by step captures most of the savings with none of that complexity.
Two implementation details that matter:
Record the model per call. Without it you can't attribute a quality regression to a routing change, and routing changes are exactly the kind of thing that gets made casually.
Keep prompts per model, not shared. A prompt tuned for a large model often underperforms on a small one. Sharing prompts across tiers is the most common reason routing "doesn't work" — the experiment was confounded.
🔍 Deciding empirically
Don't guess which steps are safe. Measure, per step:
- Run your eval suite with everything on the large model. This is the baseline.
- Move one step to the small model. Re-run. Compare pass rates for that step's assertions specifically, not the overall score — an overall score dilutes a step-level regression into noise.
- Keep the change if quality holds; revert if not.
- Repeat, one step at a time.
Tedious, and it produces a routing table you can defend. Moving several steps at once tells you nothing about which one caused the drop.
Where the savings actually are
Two things dominate, and neither is what people expect:
Call frequency, not call cost. A step running once per turn across a forty-turn run matters far more than the single expensive synthesis at the end. Look at your per-step call counts before optimizing anything.
Input tokens, not output tokens. Agent calls are input-heavy — the whole accumulated context, re-sent. A step that runs on a large context is expensive regardless of which model handles it, which points at a complementary optimization: give the small-model steps a reduced context rather than the full array. An extraction step doesn't need the conversation history.
✅ That last point is often worth more than the routing itself: same model, smaller context, most of the saving.
The takeaway
Route by step, decided by measurement rather than intuition, with per-model prompts and the model recorded on every call. Move classification, extraction, and mechanical transformation to a cheaper tier; keep tool selection, planning, and anything destructive on your strongest. And check whether the step even needs the full context first — trimming what you send is frequently the larger win, and it doesn't risk quality at all.