What Runs Locally Today and What Still Won't Soon

The local-versus-hosted decision usually gets made on ideology — data sovereignty, cost, independence — and then discovers the capability boundary afterwards. Better to know where the boundary sits, and which side of it your workload is on.

The constraints that set the boundary

Memory. The binding limit on consumer and small-server hardware. A model has to fit, alongside its context, alongside whatever else the machine is doing. This is the hardest wall and it moves in hardware generations.

Sustained throughput. A laptop can run a model; it can't run one continuously at full draw without thermal and battery consequences. Fine for occasional inference, poor for an agent loop making dozens of calls.

Effective context length. Local models typically handle shorter effective contexts than their nominal limits suggest, and agent loops are context-hungry by construction. This bites harder than raw capability for agent work specifically.

Time-to-first-token on cold start. Loading a model into memory takes real time. Acceptable for a long-running service; noticeable for something invoked occasionally.

What's comfortably local

Work that's bounded, repetitive, and doesn't need a long context:

• Embedding. Well-established, removes a per-document cost, keeps the corpus in-house. Often the single best reason to run something locally.
• Classification into a closed set. Small input, small output, high frequency.
• Field extraction against a schema. Constrained and checkable.
• Transcription and OCR, which are specialized models rather than general ones.
• Redaction and PII detection before content leaves the network — where local isn't cheaper, it's the entire point.
• Reranking retrieved candidates.

What's borderline

• Summarizing a single document. Works, quality varies by document length and domain.
• Simple code completion. Established locally; quality gap to hosted is real but the latency advantage sometimes dominates.
• Short agent loops with few tools and a small context — three or four turns, a handful of tools. Plausible, and degrades quickly as either grows.

What isn't local soon

• Wide-space tool selection. Capability-sensitive and it's the failure that's most expensive to diagnose.
• Long coherent multi-step runs. Combines capability, effective context, and sustained throughput — all three constraints at once.
• Planning under ambiguity.
• Anything needing a large context window in practice.
• Final output a person reads and judges, where the quality gap is most visible.

⚠️ The pattern: local handles the steps, hosted handles the orchestration. An agent loop running entirely locally is the hard case, not the easy one, because the loop is exactly the context-hungry, capability-sensitive, sustained-throughput part.

✅ The hybrid that works now

Local for the frequent bounded steps, hosted for the reasoning:

embed / classify / extract / redact   → local
tool selection / planning / synthesis → hosted

This captures most of the cost benefit, all of the privacy benefit for the data that matters (since redaction happens before anything leaves), and none of the capability penalty on the steps where it would hurt.

The boundary needs designing deliberately — what leaves the network, in what form, is the whole point of the arrangement, and it should be enforced in code rather than by convention.

🔍 Deciding for your own workload

• How long is a typical run? Many turns pushes toward hosted.
• How large is the context at its peak? Large pushes toward hosted.
• How wide is the tool space? Wide pushes toward hosted.
• What's the frequency? High frequency on a bounded step is the local sweet spot.
• Is there a hard privacy constraint, or a contractual one? ✅ Check precisely which — "not used for training" is satisfied by terms, "must not leave our network" is a different and much more expensive requirement, and they get conflated constantly.
• Is there a machine that's actually free to do this? Idle hardware is cheap; dedicated hardware bought for the purpose is not.

The takeaway

Local handles bounded, repetitive, short-context steps well — embedding, classification, extraction, redaction — and that's where the cost and privacy wins are. The agent loop itself is the hardest thing to localize because it stresses capability, effective context, and sustained throughput simultaneously. Run the hybrid, enforce the network boundary in code, and check which privacy requirement you actually have before committing to the expensive one.