#scaling
Every post tagged "scaling".
11 posts · showing 1–10
-
Your Agent Fleet Kept Its Own Quota Pinned for Forty Minutes: Synchronised Retries Are a Self-Inflicted Outage
The provider rejected one batch of calls; the fleet turned that into forty minutes of self-inflicted downtime. Why exponential backoff alone makes synchronised retries worse, the sub-second signal that exposes it, and the three-part fix that actually breaks the loop.
-
One Worker Pool or Two? Separating Fast Agent Runs From Slow Ones
A handful of long research runs can starve hundreds of eight-second ticket replies without a single error firing. The fix isn't more workers — it's deciding, at admission, which lane a run belongs in.
-
Graceful Degradation When the Model Provider Is Down
A fallback exercised only during incidents is exercised for the first time during an incident. Six degradation rungs, and the three cheapest ones cover most of the value.
-
Request, Worker, or Durable Execution? Hosting Models for Agents
Where a run lives decides how long it can be, whether it survives a deploy, and whether it can wait for a person. Pick by the longest wait it can hit — and keep state out of process memory so the answer can change.
-
Latency Budgets for Agents: Where the Seconds Actually Go
"Try a faster model" addresses maybe a third of the time in a typical run. Split the twelve seconds five ways and the real culprits — turn count and tool latency — are usually the ones nobody measured.
-
Pricing an Agent Product When Your Costs Are Variable
One customer can cost a hundred times another on the same plan, and they're often the ones getting the most value. Five pricing models, their failure modes, and the trap where every quality improvement cuts your margin.
-
Multi-Tenancy for Agent Products: The Boundaries That Must Hold
Data, memory, context, and capacity — four boundaries, and two of them are agent-specific enough that no existing multi-tenancy checklist mentions them. Few-shot examples harvested from production are a leak.
-
Queueing and Backpressure for Agent Workloads
A run can take four seconds or four minutes, and you can't tell in advance. Size the pool to the provider's rate limit, split queues by shape — and check the one timeout relationship that silently runs your task twice.
-
Cost Attribution: Finding Which Feature Is Eating Your Token Budget
Agent spend is dominated by a tail of expensive runs, so every averaged chart hides the cause. Five tags on each model call, one table of your twenty priciest runs, and the mystery usually resolves in an afternoon.
-
Vector Database or Postgres? Choosing Storage for Agent Memory
Most agent memory reads are filtered similarity or plain structured lookup — relational workloads wearing an embedding. Write down your ten most common retrievals before you add a second database.