TL;DR
The study shows: temperature 0 does not make an LLM's answer identical. If you run the same row-by-row task several times, some of the verdicts change. The cure is not to trust a single run: repeat the contested rows and take the majority, and save the final answers instead of recomputing them.
Here's the pain: you run the same review classification twice, and the "positive" count is 47 one time and 52 the next. Or you calculate the share of complaints, and the number in the report doesn't match yesterday's. The cause is the model call itself. The server processes parallel requests in batches, the order of computation shifts slightly, and a borderline row tips one way or the other. Even two identical requests within the same run diverge just as often as requests from different runs.
Worst of all, errors multiply with the number of rows. If the answer depends on every row (a full list, a table, all matches), then with 0.1% unstable rows and 1,560 pairs, the chance of getting the same result twice is about 21%. Counts and averages dampen some of the wobble, and "first N matches" dampens it even more. What helps is re-querying only the contested rows (cheap) and recording the verdicts you got (then the answer is reproducible).
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Run the row-by-row task 2–3 times (separate requests) → a "row × run" table
STEP 2: Find the rows where verdicts didn't match → a list of contested rows
STEP 3: Run the contested rows 3–5 more times → majority vote
STEP 4: Record the final verdicts in a table → work from it from now on, don't recompute
STEP 5: Report the result with a range, not a single number
Steps 1–3 require separate requests or simple automation (for example, a Make or n8n scenario written with an LLM's help). Everything is done with ordinary prompts.
Example in Practice
Task: A marketer at a sneaker brand is going through 400 Amazon reviews. They need to calculate the share of negative ones for a report to their manager. One run gave 18%, a second run 22%. Which number should they trust?
Prompt (step 1, identical for all runs):
You are labeling customer reviews of sneakers on Amazon.
For each review, output exactly one label: POSITIVE, NEGATIVE, or MIXED.
Negative — if the customer is unhappy with the product, sizing, quality, or delivery
and doesn't recommend buying. If there are both pros and cons — MIXED.
Answer format — table only: # | label. No explanations.
Reviews:
1. "Ordered a 9, they run small, the sole squeaks. Already filed a return."
2. "Killer value for the price, been wearing them three months, nothing's come unglued."
3. "They look awesome, but the toe cracked after two weeks."
...
Prompt (steps 2–3, reconciliation):
Below are two label tables for the same reviews from two independent runs.
List only the numbers where the labels didn't match, in the format: # | label 1 | label 2.
Don't show the rest. At the end, write the total number of mismatches.
Then you send the contested reviews 3–5 more times in new, separate chats (or requests), take the label that came up most often, and enter it in the table.
Result: After two runs you'll have label tables. Reconciliation will show a short list of contested numbers. These are usually borderline "mixed" reviews. After the vote, you'll have a final table and a negative share you can reproduce, because the verdicts are recorded. In the report, it's worth giving the number with a caveat: "about 20%, with this many contested reviews."
Why It Works
The weakness. For each row, the model doesn't produce "the one correct answer" but an answer drawn from a probability distribution. For obvious rows, the distribution is narrow and the answer is stable. For borderline rows, two options are almost equally likely. Temperature 0 picks the most likely one, but on the server, tiny computational noise can still tip the scales.
The strength. Not all rows are unstable, only a small fraction, and specifically the ones where models also disagree with each other. In other words, instability reflects genuine ambiguity in the row. That's why contested rows are easy to find: they diverge between runs.
How the method exploits this. Disagreement between runs works as an ambiguity detector. Instead of an expensive full rerun, you spend money only on the contested rows. Recording the final verdicts turns a random answer into a fixed one, and from then on you work with it, not with a new roll of the dice.
Levers: - Number of runs in step 1 (2 → 3): more runs reveal more unstable rows. The share of changing rows keeps growing with the number of runs and doesn't plateau. - Number of repeats for contested rows (3 → 5 → 9): more means a more stable majority, but costs more. - Task type: ratings on a 1–5 scale and free text change more than looking up facts stated directly in the text. Budget more repeats for scales. - What you return as the answer: counts and shares are more robust than a full list, because opposite errors cancel each other out.
Prompt Template
Task: {what_to_determine_for_each_row}.
Allowed labels: {label_list}.
Borderline rule: {how_to_decide_contested_cases}.
Answer format — table only: # | label. No explanations.
Rows:
{numbered_list_of_rows}
Run reconciliation:
Below are {number} label tables from independent runs for the same rows.
List only the numbers where the labels didn't match: # | label 1 | label 2 | ...
At the end — the number of mismatches and their share of the total rows.
{run_tables}
Fill in: what exactly is decided per row, a closed list of labels, an explicit rule for borderline cases (it narrows the ambiguity), and numbered rows. Do the runs in separate chats, otherwise the second answer will adjust itself to the first.
🚀 Quick start — paste into a chat:
Here is a template for row labeling with a stability check. Adapt it to my task: {your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what the rows are, which labels are allowed, and how to decide borderline cases. This is needed to remove ambiguity from the rule itself: the sharper the boundary, the fewer borderline rows and the fewer mismatches between runs.
Limitations
⚠️ Voting doesn't remove variability: it reduces it but doesn't zero it out. The result can still differ slightly on a full recompute.
⚠️ It depends on the model: the share of changing rows differs by two orders of magnitude across models. In the study, the reasoning model was noticeably less stable than the plain one, and commercial models vary too. You need to test your own model on your own data.
⚠️ A chat isn't a lab: in a regular chat you don't know or set the temperature. The scale of the problem there may differ from the study.
⚠️ Confidence signals require API access: predictors of "which row will flip" (the margin between the top two tokens) come from log probabilities. Chats don't expose them. The simple approach works: compare runs.
⚠️ Only text and tables were tested: images and audio weren't studied. The paper's text on voting and cascades isn't provided in full; I'm relying on the stated conclusions.
⚠️ One model can be consistently wrong: agreement between runs doesn't mean correctness. If a model has a systematic bias, it repeats it every time. For accuracy, you need a check against a labeled sample.
How They Studied It
An IBM team took three existing benchmarks for "semantic" database queries, where an LLM judges each row: SemBench, LRO-Bench, and UDA-Bench. They reran them unchanged on eight model deployments (open-weight and commercial) and recorded 1.2 million model calls, every answer for every row. Previously, benchmarks published only the final quality. Separately, the authors re-analyzed SemBench's published results: five runs of Gemini 2.5 Flash.
The findings were unexpected. At temperature 0, between 0.07% and 10.2% of rows changed across five runs, up to 24% on one task. In SemBench's published results, 21% of text cells changed in quality from run to run, even for engines with a deterministic plan. In Movie Q3 (counting positive reviews), gpt-oss-120B returned four different numbers over ten runs. Llama-3.3-70B on two different servers gave stable but different numbers: the deployment affects the result.
The main observation: it's not about "between runs." Identical requests within the same run diverge just as often. And the rows that change are the ones models argue about with each other. On UDA-Bench, a few percent of changed values altered about two-thirds of the final answers. This matched the prediction of a mathematical "possible worlds" model. The practical takeaway: one run is one roll of the dice, not a measurement.
Adaptations and Extrapolations
These are my ideas; the paper didn't test them.
🔧 Technique: a different model instead of repeats → catches systematic errors
The paper shows that unstable rows are the ones where models disagree with each other. So you can find contested rows not by repeating one model, but by running them through a second model (for example, Claude and GPT). Send the mismatches for manual review.
Below are label tables from two different models. List only the rows with a mismatch
and briefly (up to 10 words) explain what makes the row ambiguous.
Extrapolation: a rule for an agent. If an agent in Cursor or Claude Code does bulk labeling or extraction, add this to its instructions:
Always save the results of row-by-row classification and extraction to a file
(CSV). Don't recompute them on follow-up questions — read from the file.
If the task requires a share or a list, first do two independent runs,
list the mismatches, and recheck only those separately.
Resources
- Paper: Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing
- Authors: Oktie Hassanzadeh, Dharmashankar Subramanian, IBM (Yorktown Heights, NY, USA)
- Benchmarks: SemBench, LRO-Bench, UDA-Bench
- Engines: LOTUS, Palimpzest, ThalamusDB, BigQuery