#scaling
Every post tagged "scaling".
9 posts · showing 1–9
-
Graceful Degradation When the Model Provider Is Down
A fallback exercised only during incidents is exercised for the first time during an incident. Six degradation rungs, and the three cheapest ones cover most of the value.
-
Request, Worker, or Durable Execution? Hosting Models for Agents
Where a run lives decides how long it can be, whether it survives a deploy, and whether it can wait for a person. Pick by the longest wait it can hit — and keep state out of process memory so the answer can change.
-
Latency Budgets for Agents: Where the Seconds Actually Go
"Try a faster model" addresses maybe a third of the time in a typical run. Split the twelve seconds five ways and the real culprits — turn count and tool latency — are usually the ones nobody measured.
-
Pricing an Agent Product When Your Costs Are Variable
One customer can cost a hundred times another on the same plan, and they're often the ones getting the most value. Five pricing models, their failure modes, and the trap where every quality improvement cuts your margin.
-
Multi-Tenancy for Agent Products: The Boundaries That Must Hold
Data, memory, context, and capacity — four boundaries, and two of them are agent-specific enough that no existing multi-tenancy checklist mentions them. Few-shot examples harvested from production are a leak.
-
Queueing and Backpressure for Agent Workloads
A run can take four seconds or four minutes, and you can't tell in advance. Size the pool to the provider's rate limit, split queues by shape — and check the one timeout relationship that silently runs your task twice.
-
Cost Attribution: Finding Which Feature Is Eating Your Token Budget
Agent spend is dominated by a tail of expensive runs, so every averaged chart hides the cause. Five tags on each model call, one table of your twenty priciest runs, and the mystery usually resolves in an afternoon.
-
Vector Database or Postgres? Choosing Storage for Agent Memory
Most agent memory reads are filtered similarity or plain structured lookup — relational workloads wearing an embedding. Write down your ten most common retrievals before you add a second database.
-
Rate Limits Are an Agent Design Constraint, Not an Ops Problem
Backoff assumes a predictable call count. Agents burst, retry at three layers, and generate more traffic the more they're throttled — which makes rate limits an architecture question, not an ops ticket.