Why is my LLM producing inconsistent results

tl;dr

Why is my LLM producing inconsistent results? Because it rolls dice on every token — temperature sets how wild the bidding gets, and no seed means new dice every run. We asked one 2,497,280,256-byte GGUF (Qwen3-4B Q4_K_M, sha256 7485fe6f…df5) the same question five times: five different answers, one of which opened by denying the other four. Pin the seed and the reply replays byte-for-byte — and temp=0 is still not a promise: on our bench the batch size flipped a token while every knob said deterministic.

The same prompt gave five different answers because sampling picks each token from a probability list at temperature > 0, and with no seed every run loads a fresh roll. Set an identical seed (OpenAI-compat seed:, ollama options.seed, llama.cpp --seed) and the answer replays exactly — 3/3 byte-identical at temperature 0.7 on our bench. But even greedy decoding drifts: with temp=0 and the seed pinned, changing the batch size (-ub 512 → -ub 8192) flipped a token — “It is” became “It’s”. Same-model A/B tests need the same seed and the same batch shape.

Fig 1 — five runs, five answers, one self-contradiction. Run 2 (dim) is the receipt: same file, same box, same prompt, opposite opening.

Why does my LLM give a different answer every time?

seed — a saved dice roll. Every sampling choice starts from one number; set it and the whole shuffle replays. Omit it and the engine draws a fresh one per request.

temperature — how wildly the model bids. Each next token is an auction: the model holds a probability list, temperature reshapes it, one token wins the draw. 0.7 is ordinary dice; 0 means “take the favourite every time” (greedy).

top_p — a second, stricter filter. It trims the candidate list before the roll. It narrows the dice; it does not pin them.

The model never “has” an answer. For every single token slot it holds a probability list, temperature reshapes the list, and one token wins the draw. The winner is appended, and the next auction opens with a different list — because the text it is continuing is now different. One early divergence compounds: run 2 said “not” where runs 1, 3, 4 and 5 said “Python”, and from that single character the five replies live in different universes. Five runs are five independent draws, not one answer printed five times.

How do I make my LLM replies repeat exactly?

Name the roll. We re-ran the identical ask three times with seed: 42 at temperature 0.7 — the not-deterministic setting — and got 3/3 byte-identical replies (generation speed wobbled 6.18 → 6.74 → 7.63 tok/s; the words did not wobble at all). The seed pins the dice even while they are being thrown.

curl http://localhost:8080/v1/chat/completions -d '{
  "model": "qwen3:4b",
  "messages": [{"role": "user", "content": "Explain GGUF quantization"}],
  "temperature": 0.7,
  "seed": 42
}'

verify: same prompt + same seed → byte-identical reply, 3/3 on the bench (temp 0.7, not just greedy) · det-results-a.json

The same knob exists everywhere you already work: options.seed in ollama’s /api/generate, seed in the OpenAI-compat chat lane, --seed on the llama.cpp command line. ollama’s runner is llama-server wearing a jacket, so the receipt below the hood is the same one — the seed reaches the same sampler.

Why is temperature 0 still not deterministic?

Because greedy decoding removes the dice. It does not remove the rounding. Inside the network every token score is thousands of additions, and floating-point addition is not associative: add a long column left-to-right, then add it in chunks — same numbers, possibly a different last digit. When the top two candidates sit close enough, the rounding is the decision. And the summation order depends on how much work the batch does at once.

So we pinned everything — temp=0, seed=42, same server, same prompt — and changed exactly one flag, the micro-batch width:

ask	-ub	seed	output opens	vs -ub 512
1–2	512	42	“ Python. It is used in many…”	baseline, 2/2 identical
1	8192	42	“ Python. It’s used in many…”	FLIPPED — one token
2–3	8192	42	“ Python. It is used in many…”	snapped back, 2/2 baseline

The loaded die, live: every knob said deterministic, the batch shape moved one token anyway — and only on the first ask after boot.

Read the third row twice. The flip appeared once, on the cold server’s first ask, and the next two asks at the identical settings returned the baseline text. Warm-server state — the prompt cache, the slot layout — joins the summation order as things a “deterministic” reply depends on. We caught the flip; we will not pretend we caught the culprit. The dice are loaded. The hand stays hidden.

What does run-to-run variance cost?

The bill, five times over. We asked for a hello in exactly three words, five times, free seed, 256-token budget. The model answered zero of five — every run spent the entire budget thinking before the cap cut it off, and the thinking itself wobbled: 884 · 954 · 986 · 915 · 885 characters of it for the identical ask.

Fig 2 — the wobble is the point: a 102-character spread on a fixed ask. The thinking bill from October’s LFM3 post, five times in a row.

We asked for hello in three words and received 884 to 986 characters of soul-searching and no hello. The model is not indecisive about its answer — it is indecisive about everything before the answer, and you pay for that whether or not the answer arrives. Any “exact answer” ask is a budget: cap the thinking (think: false, or the model’s own off-switch) or the cap eats you first.

The fix list

Pin the seed everywhere you compare: seed (OpenAI-compat) · options.seed (ollama) · --seed (llama.cpp)
A/B tests: same seed, same flags, warm server — and re-run the pair. Never trust one lap (see the -ub 8192 row: cold ask flipped, warm asks matched)
Temp 0 for extraction and evals — then diff the outputs anyway; the batch shape can flip a token with the dice out of the picture
Cap the thinking on “exact answer” asks, or the budget dies before the answer ships (0/5 on our hello)
Grade on N ≥ 3 runs and read the median — any single run is one dice throw, not the model

symptom	what is happening	the fix
different answer every run	no seed; temp > 0 reshapes the list per draw	pin `seed`; the reply replays byte-for-byte
different answers at temp=0	batch shape / warm-server state changed the summation order	same `-ub`/`-b`, same engine build, warm before you measure
the answer never arrives	thinking ate the token budget (0/5 here)	cap or disable thinking; read `completion_tokens`, not vibes

receipt · run 2 denies runs 1, 3, 4, 5 (same prompt, same box, no seed)

run 1: “ Python. Python is widely used in data science, web…”
run 2: “not Python, Java, or C++. It’s actually a language that’s
       not even a progr…”
run 3: “ Python.  But there are other languages that are…”
run 4: “ Python. But why? Python is a high-level, interpreted…”
run 5: “ Python. Python has a lot of applications in web…”

Four runs said Python. One run opened by disputing the premise. Same weights, same box, same string in. One of the five is wrong about being a deterministic machine; the other four are merely lucky.

last tested 2026-11-11

Qwen3-4B Q4_K_M · 2,497,280,256 B · sha256 7485fe6f…df5 · llama-server 0.9.0-dev (b12110) · one fresh Arch VM, 4 threads — CPU bench, no VRAM claims