The Cost of Choosing Your Model Too Early

Teams pick a model at the start of a project and then build around its particular behavior. Six months later the choice is load-bearing in ways nobody intended, and switching — to a cheaper tier, a different provider, or a newer version — means unpicking accumulated compensations.

The choice itself is rarely the mistake. Failing to keep it reversible is.

How the coupling accumulates

Prompts absorb the model's quirks. A line added because this model kept forgetting something. A formatting reminder for a behavior it exhibits. Each is a reasonable local fix; together they're a prompt tuned to one model's tendencies, and it will underperform on any other.

The harness compensates for its weaknesses. Extra validation because this model produces malformed output occasionally. A retry because it sometimes stops early. Those layers stay after the weakness is gone, and nobody remembers which layer exists for which reason.

Behavior gets encoded in the eval thresholds. "Passes 9/10" was calibrated against one model's variance. Switch and the thresholds mean something different.

Assumptions leak into the design. Context budgets sized to one window, step decomposition shaped by how far this model reliably chains, tool granularity tuned to its selection accuracy.

⚠️ None of this is visible as coupling. It looks like a working system, and the cost only appears when you try to change something.

Keeping the choice reversible

One place that names the model. Configuration, not scattered through call sites. Obvious and frequently violated once a second model gets added for one step.

Per-model prompts from the start, even when you only have one. A prompts/{model_family}/ structure costs nothing when there's one entry and means a second model is an addition rather than a refactor.

Record the model on every call. Without it you cannot attribute behavior to a model change, which makes any migration an unmeasurable leap.

Annotate compensations. ✅ When you add a workaround for a model's behavior, say so in a comment: # gpt-family sometimes emits trailing prose after JSON — strip it. Then a future migration has a list of things to test removing, instead of a prompt nobody dares touch.

Keep a frozen eval set with recorded tool responses. This is what makes switching a measurement rather than a gamble. Without it, "is the new model better for us?" is unanswerable.

💡 Deferring the choice where you can

Some decisions don't need making early at all:

• Tier per step can be decided after you have usage data. Start uniform, instrument, route later.
• Provider can stay abstracted behind your own thin adapter — not a heavyweight framework, just a function you own that takes messages and tools and returns a reply.
• Fine-tuning is almost always premature early. It couples you to a specific model, costs re-work on every migration, and usually solves a problem better prompts or better tools would have.

→ The general principle: make the choice that's cheapest to reverse, and defer the ones that aren't until you have evidence.

The counter-argument, honestly

Deferring has costs too. Abstraction layers add indirection and can obscure provider-specific features worth using. A team that spends its early weeks building model-agnostic infrastructure instead of shipping has optimized for a flexibility it may never exercise.

The proportionate version: a thin seam you own, per-model prompt directories, annotated compensations, and a frozen eval set. That's maybe a day of work and it preserves optionality without building a platform. Anything heavier is premature.

🔍 Checking your current coupling

• Can you name the model in one place?
• Could you run your eval set against a different model this afternoon?
• Do you know which prompt lines exist because of model quirks?
• Do you know your cost and quality per step, so you'd notice a regression?

Four questions. If the answers are no, the model choice is more load-bearing than anyone decided it should be — and the fix is cheap now and expensive later.

The takeaway

Picking a model early is fine; letting the choice become irreversible isn't. Keep one place that names it, prompts organized per model family, workarounds annotated with why they exist, and a frozen eval set that makes switching measurable. That's a day of work, it costs nothing while you don't need it, and it's the difference between a migration and an excavation.