Why Two Identical Runs Give Different Answers

You run the same agent on the same input twice and get two different results. Not wildly different — one used three tool calls, the other four; one mentioned a caveat the other didn't. It's unsettling if you're used to software that does the same thing every time, and the first instinct is to look for a bug.

There are usually four sources of the variation, and only one of them is a bug. Knowing which is which saves a lot of confused debugging.

1. Sampling — the model chooses among possibilities

A model doesn't compute one answer; it produces a probability distribution over next tokens and one gets selected. The temperature setting controls how much the selection favours the most likely option.

At temperature 0, the most likely token is always chosen, which sounds deterministic and mostly is — but not entirely. Two tokens can be nearly tied, and tiny numerical differences decide it. Those differences come from the next item.

2. Infrastructure — the same arithmetic, in a different order

Providers batch requests together for efficiency, and the composition of a batch varies with what other traffic is arriving. Floating-point arithmetic isn't associative — adding the same numbers in a different order can produce a very slightly different result — so identical input can produce marginally different probabilities depending on batching, hardware, and routing.

Usually invisible. Occasionally it's enough to flip a near-tie, and then the run takes a different path from that token onward.

⚠️ The practical consequence: temperature 0 reduces variation a great deal but does not guarantee identical output. Any system built on the assumption of exact reproducibility will eventually be surprised.

3. The input wasn't actually identical

This is where most apparent model nondeterminism turns out to live, and it's the first thing to check:

• A timestamp in the system prompt, or a relative date like "today."
• Tool results that genuinely differ — a search index updated, a record changed, a list returned in a different order.
• Retrieval returning a different set because scores were close.
• A tool list built from an unordered collection, serialized differently each time.
• A session or run ID appearing somewhere in the prompt.

Before concluding the model is varying, diff the two raw arrays. → In practice they're different more often than not, and the difference is usually something small and boring in the assembly code.

4. A real bug

Shared mutable state between runs, a race in parallel tool calls, a cache returning stale data for one run and fresh for another. These produce variation too, and they're the only category worth fixing as a defect.

The tell: variation that correlates with concurrency or with ordering rather than being randomly distributed. If two runs differ only when they overlap in time, it isn't the model.

What to do about it

Accept path variation; refuse outcome variation. Two runs taking different routes to the same correct answer is normal and healthy. Two runs producing different answers is a problem — and the way to separate them is to assert on outcomes and invariants rather than on exact sequences.

Pin what you can. Explicit model version, temperature set deliberately, clock injected, tool lists in a stable order. This removes the variation you control so what's left is genuinely the model.

Measure instead of sampling once. Run an important case ten times and look at the distribution. One run tells you what happened once; ten tell you what your system does.

Design for it in the product. If users will see the output, a system that answers slightly differently each time needs either an explanation or a stored result. Re-generating an answer someone has already seen and quoted is its own category of problem.

💡 When you actually want the variation

Worth remembering that it isn't only a cost. Sampling several independent attempts and comparing them is a real quality technique — agreement is evidence, disagreement flags a hard case. Nondeterminism is what makes that possible, and a perfectly deterministic model would give you one confident answer with no signal about how confident to be.

The takeaway

Four sources: sampling, infrastructure arithmetic, an input that wasn't really identical, and genuine bugs. Check the third first — diff the raw arrays, since that's where most surprises hide. Pin the model version, the clock, and the tool ordering to eliminate what you control. Then treat path variation as expected and outcome variation as the thing to test for, and the remaining nondeterminism becomes a property to design around rather than a mystery to chase.