Why LLMs give different answers at temperature 0
Setting temperature to 0 should make a model pick the most likely token every time, so the same prompt should always give the same answer, but in practice it often doesn't. The common explanation is GPU parallelism plus floating point rounding, and that is part of the story, but research published in 2025 showed that the main cause on inference servers is that results change with the batch size, which depends on how many other requests the server is handling at that moment. For your application, assume outputs will vary: pin model versions, validate outputs, evaluate over several runs, and cache answers where repeatability matters.
Problem
Ask a chatbot the same question twice and you get two different answers. That is expected with normal settings, because the model samples tokens at random according to their probabilities, as the sampling page explains.
The surprise is that it still happens at temperature 0. At temperature 0, the model should always take the single most likely token, so the whole process should be deterministic: same input, same output, bit for bit. In practice, run the same prompt many times through an API at temperature 0, and you will often see small differences, a different word here, a different sentence structure there, and once one token differs, everything after it can differ too, because each token depends on the ones before it.
Why it matters
For a casual chat, a slightly different answer doesn't matter. For engineering, it causes real problems.
- Debugging is harder. You can't reliably reproduce the exact output a user got.
- Evaluation is noisier. If the same test case passes on one run and fails on the next, a small improvement from a prompt change can disappear in the noise.
- A/B tests need more samples, because some of the difference between two versions is random.
- Agents drift. In an agent or a coding assistant, each output becomes part of the input for the next step, so a small difference early on can lead the whole run somewhere else.
- Trust suffers. Users, auditors and regulators expect the same question to get the same answer, especially in finance, law and healthcare.
Common explanation: floating point and parallelism
Models do their math with floating point numbers, numbers with decimals, stored with limited precision. Floating point math has a property that surprises people: addition is not associative. In normal math, (a + b) + c equals a + (b + c), but with floating point numbers, the order of operations changes the rounding, and therefore the result.
a, b, c = 0.1, 1e20, -1e20
print((a + b) + c)
print(a + (b + c))
The first line prints 0.0 and the second prints 0.1, from the same three numbers.
GPUs do thousands of operations in parallel. The common explanation is that because parallel threads can finish in a different order on each run, the numbers get added in a different order, the rounding is slightly different, and that tiny difference flips which token is the most likely at some step.
Why that explanation is incomplete
In September 2025, Horace He and colleagues at Thinking Machines Lab published a detailed investigation called Defeating Nondeterminism in LLM Inference. They pointed out that floating point non-associativity is real, but the GPU explanation doesn't hold up on its own. If you run the same matrix multiplication on the same data on a GPU many times, you get bitwise identical results every time. Most of the operations in an LLM's forward pass don't depend on the unpredictable thread ordering the explanation assumes.
So something else was changing between runs.
Real cause: batch invariance
Inference servers don't process one request at a time. To use the GPU efficiently, they group many users' requests into a batch and process them together. The size of that batch depends on how many requests arrive at that moment, which you can't control and can't see.
The investigation found that many of the GPU kernels used for operations like matrix multiplication, normalisation and attention choose different strategies for different batch sizes, splitting the work in different ways for speed. Different splits mean the numbers are added in a different order, which means slightly different results. Each kernel is deterministic for a given batch size, but your request's numbers depend on what batch it lands in.
From your side, the server looks nondeterministic, because the load on the server changes every second. They called the missing property batch invariance: results that don't depend on the batch size.
What the experiment showed
They ran the same prompt, "Tell me about Richard Feynman", 1000 times at temperature 0 on Qwen3-235B, generating 1000 tokens each time. They got 80 different completions. All 1000 matched for the first 102 tokens, and at token 103 they split: 992 said Feynman was born in "Queens, New York" and 8 said "New York City".
Then they replaced the key operations with batch-invariant versions, and all 1000 completions were identical.
The fix had a cost. In their test, generation that took 26 seconds with the default setup took 55 seconds with an unoptimised deterministic version and 42 seconds after improving the attention kernel. That is why providers don't simply switch it on everywhere.
Other reasons outputs change
Batch effects explain variation within one model on one server. In a real application, there are more reasons the same prompt gives a different answer over time:
- The provider updates the model behind a name, so the model you called last month isn't exactly the model you call today.
- Requests are served from different hardware or regions.
- Your prompt changed slightly, a timestamp, a user name or the order of retrieved chunks.
- Retrieval returned different documents because the index was updated.
- A tool returned different data, like a search result or a price.
When you investigate a changed answer, check these first, because they are more common than numerical effects.
What to do in production
You can't make a hosted model perfectly deterministic, so design the application to work with variation.
Pin the model version
Use a dated or versioned model name instead of an alias that always points to the latest model, and upgrade on purpose, after running your evaluations.
Use low temperature and a seed where it helps
For extraction, classification and code, use temperature 0 or close to it. If the API supports a seed, set it. Both reduce variation, and neither guarantees identical output.
Make the output checkable
Ask for structured output and validate it. Two answers with different wording can both be correct, so check the fields and facts that matter, not the exact text.
Evaluate over several runs
Run each test case several times and look at the pass rate, not a single pass or fail. A prompt change is only an improvement if it improves the pass rate by more than the run-to-run noise.
Cache where repeatability matters
If the same question must always get the same answer, store the first approved answer and return it for identical questions, instead of generating it again. For questions that are similar but not identical, semantic caching can reuse answers to questions with the same meaning, carefully, because similar questions don't always have the same answer.
Log everything
Log the model version, the full prompt, the settings and the output for every call, so that when a user reports a strange answer, you can see exactly what was sent and what came back.
Self-host when you need bitwise reproducibility
If your use case requires the same output bit for bit, for example for audits or scientific reproducibility, run an open model on your own infrastructure with deterministic settings and batch-invariant kernels, and accept the speed cost. Inference engines have started adding options based on this research, so check what the current version of yours supports.
Nondeterminism is not hallucination
These are two different problems. Nondeterminism means you get different answers to the same input. Hallucination means you get a wrong answer, which can happen even when the output is perfectly consistent. A model can be deterministic and consistently wrong, or nondeterministic and correct every time in different words. See hallucination.
From the floor
This topic was the first session of a series I started in the July cohort called the AI blackbox, where we take one thing happening behind the scenes of an AI system and go through it in depth. We read the Thinking Machines article together, starting with the hypothesis: the problem, the common explanation, and the floating point non-associativity example, run live in Python.
One builder, who had worked in fraud analytics, asked how anyone could use LLMs in production if outputs vary like this. They said they would never use an LLM where a 0.2% error means missing a fraudulent transaction, and for that kind of decision, traditional machine learning models with tested, stable behaviour are usually the better tool. For the parts where an LLM does fit, the answer is the list above: design the system so variation is caught by validation, evaluation and people reviewing the cases that matter.
Another builder asked whether fine-tuning could fix it. It can't, because the variation comes from how the computation runs during inference, not from what the model learned, so a fine-tuned model running on the same servers has the same behaviour.