Why LLMs Give a Different Answer Every Time You Run Them

Ask a language model to finish “The cat sat on the…” three times. You might get mat, then sofa, then windowsill. The prompt did not change. The model did not “forget.” Each run is a fresh draw from a probability distribution, one token at a time.

This is the most common surprise people hit when they leave the chat box and start calling an API. A calculator returns the same number. A SQL query returns the same rows. An LLM is neither. It is a next-token sampler.

You will leave with: a clear picture of logits, softmax, temperature, and the cascade that turns one lucky first word into a whole different sentence — plus the knobs that make answers more repeatable when you actually need that.

1. Models do not retrieve a stored sentence

A search index can return the same document twice. A language model does not have a folder labeled answers/the-cat-sat-on-the.txt. During training it saw millions of sentences and adjusted billions of weights so that, given some context, it can score every token in its vocabulary.

At run time the only job is: given the tokens so far, produce a score for what comes next. Then pick one. Then do it again. That loop is called autoregressive generation.

The model is not choosing a paragraph. It is choosing the next crumb, then the next, using its own crumbs as the new path.

That is why a sentence completion is a good teaching example. There is no single correct next word. “Mat” is common. “Sofa” is fine. “Windowsill” is a bit more specific. “Quantum” is almost never licensed by the context. The model encodes that ranking as numbers, not as one locked-in string.

2. One step: logits, softmax, sample

Inside one generation step, four things happen:

1. Context — the prompt, plus any tokens already written.
2. Logits — a raw score for every token in the vocabulary (tens of thousands of numbers).
3. Softmax — those scores become probabilities that add up to 1.
4. Sample — the runtime draws one token from that distribution (or, if you force it, takes the single highest).

Softmax is the only formula you need:

$$P(i) = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}$$

$z_i$ is the logit for token $i$. $T$ is temperature. When $T = 1$, you get the model’s native distribution. When $T$ gets small, the biggest $z_i$ dominates. When $T$ gets large, the list flattens and weaker tokens get a real chance.

If the runtime always picked the top score — greedy decoding, also called argmax — every run of the same prompt would start with “mat.” Chat products almost never do that. They sample, so 42% of first tokens might be “mat,” 18% “sofa,” and so on. You clicked Generate again. You drew again.

3. The cascade: one different token, a different story

Sampling would already make answers vary. Autoregression makes the variation explode.

Suppose run 1 draws mat. The new context is “The cat sat on the mat”. The next distribution is now about what happens on a mat: sun, nap, kitchen floor. Suppose run 2 draws sofa. The next distribution is about living rooms, remotes, doorbells. The second token is not chosen from the original list. It is chosen from a new list that the first token created.

That is the cascade:

• Token 1 is a fork.
• Token 2 is a fork from that fork.
• By token 20 you are in a completely different paragraph, even though both paths started from the same eight words.

People describe this as “the model is being creative.” Mechanically it is compounding chance. A 18% first-token draw is enough to send the rest of the completion down a different hallway.

Two completions can both be fluent and still share almost no words. That is not a contradiction. They are two legal walks through the same probability landscape.

4. Temperature, top-k, and top-p

APIs expose a few knobs that decide how wild that walk is. None of them change the trained weights. They only change how the next token is chosen from the logits you already have.

Temperature

Think of temperature as how greedy the sampler is allowed to be.

At $T \to 0$, softmax collapses onto the single largest logit. That is why people say “set temperature to zero for determinism.” It is the right first move. It is not a mathematical guarantee on every GPU, which we will get to.

Top-k

Top-k keeps only the $k$ highest-probability tokens and samples among those. k = 1 is greedy. k = 40 is a common default: the tail of the vocabulary (the 50,000th most likely token) cannot sneak in.

Top-p (nucleus sampling)

Top-p is smarter than a fixed $k$. It keeps the smallest set of tokens whose probabilities add up to $p$. If the model is very sure, that set might be two tokens. If the model is torn, the set grows. p = 0.9 is a typical chat setting: ignore the weird 10% tail, sample from the plausible 90%.

Chat UIs usually combine a moderate temperature with top-p. That mix is why hitting Enter twice on a sentence completion rarely reprints the first answer.

5. A tiny sampler you can run

You do not need a 70-billion-parameter model to feel this. Fake a vocabulary of four completions and draw from it. Run the file twice.

import numpy as np

tokens = ["mat", "sofa", "windowsill", "counter"]
logits = np.array([2.4, 1.5, 1.0, 0.7])  # made-up scores
temperature = 0.8

scaled = logits / temperature
probs = np.exp(scaled - scaled.max())
probs = probs / probs.sum()

print("probabilities:", dict(zip(tokens, probs.round(3))))
print("draw:", np.random.choice(tokens, p=probs))

First run might print mat. Second run might print sofa. The scores never moved. The random number generator did. That is the whole mystery of “why is my completion different?” in ten lines.

Now set temperature = 0.01 and run it ten times. You will almost always get mat. The model did not become smarter. The lottery became boring.

Pin the generator if you want the same boring lottery every process start:

rng = np.random.default_rng(seed=42)
print(rng.choice(tokens, p=probs))

Hosted APIs expose a similar idea as a seed parameter. Some honor it. Some only honor it on a given backend version. Treat it as a best-effort replay button, not a checksum.

6. Why temperature 0 is still not always identical

On paper, greedy decoding is deterministic: always take $\arg\max_i z_i$. In production you still see drift. The model is not “ignoring” $T = 0$. The logits themselves can move by a tiny amount between calls.

Common reasons:

• GPU math is not bit-identical. Parallel reductions add floating-point numbers in a different order. A $10^{-6}$ change can swap two almost-tied tokens.
• Batching. Your request may share a kernel with someone else’s. Different batch shapes, different kernels, slightly different scores.
• Mixture-of-experts routing. Some large models send tokens to different expert sub-networks. A small numeric difference can pick a different expert, then a different next word.
• The API is not your laptop. Providers load-balance across chips, update serving stacks, and sometimes wrap your prompt with a hidden system preamble that they can change.
• You did not actually send the same prompt. A trailing space, a different system message, or chat history you forgot about is a new context, so it is a new distribution.

Researchers have shown that even greedy decoding on the same GPU can disagree across runs unless the serving stack is locked down. If your product needs string-equal output, do not bet the farm on temperature=0 alone.

Temperature 0 means “do not add extra randomness on purpose.” It does not mean “the hardware will reprint this byte for byte tomorrow.”

7. When variation is a feature, and when it is a bug

Different answers are not automatically a defect. They are a defect when the task has one correct string.

Sentence completion sits on the “wanted” side. There is no canonical cat. If you are building a product that looks like sentence completion but is actually “return the customer’s order status,” treat variation as a bug and design around it.

8. How to get the same answer twice

You cannot make a large hosted model as deterministic as 2 + 2. You can get close enough for most products.

1. Lower temperature — start at 0 for extraction, routing, and code. Keep chat at 0.7 if you want it to sound like a person.
2. Tighten the nucleus — top_p = 0.1 or a small top_k when you want the model to stay near the mode.
3. Send a seed if the vendor documents one. Re-check after they ship a new model snapshot.
4. Constrain the output — JSON schema, enum fields, regex, or a grammar. If the only legal completion is {"status":"delivered"}, wording drift has nowhere to hide.
5. Do not ask the model to invent a format. Give the exact keys. Ask for the value.
6. Cache — if the prompt is literally identical, return the previous completion. This is the only way to get byte-identical replies on a public API.
7. Judge meaning, not strings — in tests, assert that the JSON parses and the field equals 1042. Do not snapshot the whole essay.
8. Move the decision out of the model when you can. Classify the intent, then fill a template you wrote. Templates do not sample.

A useful rule for production: let the model choose among a short list you defined. “Which tool?” and “which status?” are sampling problems you can live with. “Write any English paragraph and I will parse it later” is how dashboards start lying.

completion = client.chat.completions.create(
    model="gpt-4o-mini",
    temperature=0,
    seed=42,  # honored by some APIs, ignored by others
    messages=[
        {
            "role": "system",
            "content": "Reply with one word: mat, sofa, windowsill, or counter.",
        },
        {"role": "user", "content": "The cat sat on the"},
    ],
)

That call is still sampling, but the legal set is four tokens. You turned an open-ended story into a classification. Variation drops because there is almost nothing left to vary.

9. What this is not

A few nearby explanations get mixed in. They are real, just not the main reason your sentence completion changed.

• Hallucination is a wrong claim presented as fact. A new, still-true completion of a story is not a hallucination.
• Fine-tuning changes the weights, so the distribution changes. It does not remove sampling unless you also change decoding.
• RAG (retrieval) can pin facts to a document. The model can still paraphrase the document differently each run.
• A new model version will change answers even at temperature 0, because the logits themselves are different.

If you need one more mental model: training learns a landscape. Decoding walks it. Temperature and top-p choose how much the walker is allowed to wander. Your prompt chooses which hill you start on. Clicking Generate again starts a new walk from the same hill.

Takeaways

• An LLM does not fetch a stored sentence. It samples the next token, then conditions on that token, then samples again.
• Different runs are different draws from the same distribution. The first unlucky or lucky token can rewrite the rest of the paragraph.
• Temperature, top-k, and top-p only reshape that distribution. They do not retrain the model.
• temperature=0 removes intentional randomness. GPU math, batching, and API internals can still move a tie.
• Want variety for writing. Want a short allowed list, a schema, a cache, or a template when the answer has to be the same.

Finish the sentence yourself, then finish it again. If both endings sound like English, the model is doing what it was built to do. If your product needed one ending, stop asking it to write a story — ask it to pick from a list you control.