Local Models for Agent Work: Where They Actually Fit
"Can a local model replace the frontier one?" is the wrong question. Which steps can move is the right one — and the comparison people skip is against a small hosted model, where most of the saving already lives.
The case for running models on your own hardware is usually made on cost or privacy and then evaluated on whether a local model can replace the frontier one for the whole agent. That framing produces a disappointing answer.
The productive question is narrower: which steps of an agent run can move locally, and what does that buy?
What local models do well in an agent
Classification with a closed label set. Intent routing, ticket categorization, deciding which of five paths a request takes. Local models handle this well, and it's often the highest-frequency call in a system.
Structured extraction against a schema. Pulling fields from a document. Constrained by the schema and verifiable in code, so a mistake is caught rather than propagated.
Embedding. Running embeddings locally is well-established, removes a per-document cost, and keeps the corpus in-house. For many teams this alone justifies the setup.
Filtering and triage before an expensive call. A local model deciding whether something needs the frontier model at all — a cheap gate in front of an expensive step.
Redaction and PII detection before content leaves your network. Notable because it's a case where local isn't just cheaper, it's the point.
Where they struggle in agent loops specifically
Tool selection across a wide surface. This degrades sharply with model capability, and the resulting failures are expensive and mis-attributed — they look like tool-description problems.
Long-context reasoning. Agent contexts get large. Effective long-context performance is where the gap remains widest, and it's exactly what a multi-turn run demands.
Multi-step planning under ambiguity. The capability you're paying the frontier model for.
Reliable structured output at length. Improving, but a local model producing a long structured document is more likely to lose the format partway, and the failure mode is a partial parse rather than a clean error.
The honest cost picture
Local isn't free, and the comparison is often made against the wrong baseline.
Real costs: hardware or dedicated instances, which cost the same whether idle or busy. Operational work — serving, scaling, upgrades, monitoring. Engineering time to evaluate and re-evaluate. And the opportunity cost of the team's attention.
Where the arithmetic works: high, steady volume on a narrow task. A classifier running millions of times a month on predictable input is a good candidate — the hardware is saturated and the task is one a small model does well.
Where it doesn't: bursty or low volume, where you're paying for idle capacity, and heterogeneous tasks where you'd need several models and the operational burden multiplies.
⚠️ The comparison that's usually skipped: a small hosted model. Much of the cost benefit people attribute to local comes from using a smaller model, and that's available without operating anything.
When privacy is the actual driver
Different calculus entirely. If data genuinely cannot leave your network — regulation, contract, classification — then local isn't an optimization, it's the requirement, and the question becomes what's achievable within it.
The hybrid that works: local models handle anything touching the restricted data, producing redacted or aggregated output; hosted models handle reasoning over the redacted results. This needs a clear boundary and a mechanism enforcing it, but it lets you use frontier capability without exposing raw data.
✅ Be precise about what the requirement actually says. "Data cannot be used for training" is satisfied by contractual terms with hosted providers. "Data cannot leave our network" is a genuinely different constraint. These get conflated, and one of them is much cheaper to satisfy.
🔍 Evaluating a candidate step
Same method as any model routing decision:
- Establish the baseline with the frontier model on your eval suite.
- Move one step locally.
- Compare that step's assertions specifically, not the overall score.
- Measure latency too — a local model on saturated hardware can be slower than a hosted call.
Write prompts for the local model rather than porting the frontier one. Sharing prompts across capability tiers is the most common reason these experiments produce a falsely negative result.
The takeaway
Ask which steps can move, not whether the whole agent can. Classification, extraction, embedding, filtering, and redaction are good candidates; tool selection, planning, and long-context reasoning generally aren't. Compare against a small hosted model before comparing against a frontier one — that's often where the real saving is, with none of the operations. And if privacy is the driver, check precisely which constraint you're under, because the two common versions have very different costs.