TL;DR
When you ask an LLM to score several candidate answers (0–100) so you can pick the best one, the method works far worse than it looks. Instead, ask directly: "Which is better — A or B?" Sounds like a small thing, but the difference is huge.
The root of the problem: numeric scores produce too few distinct values (~20 levels out of 100), so 67% of comparisons end in a tie — the model gives both candidates the same score, and the choice becomes random. The global correlation looks decent (r = 0.47), but that's an illusion — the metric is propped up by the obvious cases, and on real tasks the judge captures only 21% of what a perfect choice would deliver.
Pairwise comparison ("A, B, or tie?") cuts ties from 60% to 4%, and the value of the choice rises from 21% to 61%. The model doesn't get smarter — it just stops "abstaining" through identical scores and is forced to make a decision.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
OLD WAY (broken):
Request → score A and B on a 0-100 scale
→ A: 72, B: 72 → tie → random pick
→ Value: 21%
NEW WAY (works):
Request → which is better: A or B?
→ explicit choice + reason
→ Value: 61%
If there are more than two candidates → pairs in turn (A vs B, winner vs C...)
Example
Task: You've written three versions of the pitch for your real-estate newsletter on Substack and want to keep the best one.
Prompt (pairwise comparison):
Evaluate two descriptions of a real-estate newsletter. Decide
which one is better at hooking a potential subscriber.
Description A:
"A newsletter about the Chicago real estate market. Analysis, trends, tips
for buyers and investors."
Description B:
"I help you avoid losing money on real estate: I break down
specific properties, explain the risks and calculate the real return."
Which is better for a subscriber — A, B, or are they equal?
First explain your reasoning, then give a clear verdict: A / B / TIE.
Result: The model first shows its reasoning (why one is more specific, why the other is vaguer), then delivers an unambiguous verdict. If you have three versions, compare the first two first, then pit the winner against the third.
Why It Works
The problem with numeric scores. When you ask for "a score from 0 to 100," the model thinks in coarse buckets: "good" → 70–80, "very good" → 85–90. When comparing two solid candidates, both land in the same bucket, get the same score — and the choice becomes a coin toss. That's not the model being lazy; it's a structural problem with the numeric format.
Why pairwise comparison works. When you ask "A or B?", the model can't escape into a tie through identical numbers. It's forced to find at least some difference. And, importantly, it does see the difference — the numeric scale just didn't let it express it. In 97.9% of cases where numeric scores produced a tie, pairwise comparison found a winner and picked correctly 78% of the time.
Control knobs: - "A / B / TIE" — keep TIE so the model doesn't invent a winner where things really are equal - "explain your reasoning" — put it before the verdict so you can see the logic and argue with it - Confidence — after the verdict, add "rate your confidence from 1 to 5" → if the model says 2, the candidates really are equivalent - Many candidates → don't run every pair at once, run a tournament: the winner of pair 1 faces the winner of pair 2
Prompt Template
Evaluate two versions of {what_we_compare}.
Version A:
{version_A}
Version B:
{version_B}
Selection criterion: {criterion}
First explain how they differ on this criterion.
Then give your verdict: A / B / TIE.
Rate your confidence: 1 (nearly equal) — 5 (clear winner).
What to fill in:
- {what_we_compare} → an offer, an email subject line, a product description, a support reply
- {criterion} → conversion, clarity, persuasiveness, fit with the brand voice
- {version_A} and {version_B} → your full texts
🚀 Quick start — paste into a chat:
I want to compare several versions of {what you have}
and pick the best one using pairwise comparison.
Adapt this template to my task and ask me clarifying questions:
[paste the template above]
The LLM will ask what you're comparing, by which criterion, and how many candidates there are — because pairwise comparison needs to be tuned to the specific task and number of candidates.
Limitations
⚠️ Only works when there are real differences: If the candidates are nearly identical, even pairwise comparison gives an unreliable result (accuracy drops from 94% on clearly different candidates to 66% on near-identical ones).
⚠️ Doesn't scale linearly: For 4+ candidates you need tournament pairs (6 comparisons for 4 candidates). More requests, but a more reliable decision.
⚠️ Calibration won't fix the quality metrics: The author checked — "adjusting" numeric scores through calibration barely helps (+2.6% recovery). The problem is the format, not the numbers.
⚠️ Doesn't replace objective evaluation: If there's a correct answer (math, code), check it directly rather than through a judge.
How It Was Studied
The researcher took 5,000 prompts from Chatbot Arena — real user questions — and prepared 4 candidate answers of varying quality for each. GPT-5 was appointed judge and asked to score each answer on a 0 to 100 scale. Then the question was: how often does "best by score" match the actually best answer?
The result was surprising. The global correlation between the judge's scores and real quality looked tolerable (r = 0.47), but when they measured what actually matters — how much the judge improved the choice compared to random — the number was dismal: 21%. In other words, the judge delivered only a fifth of the value of a perfect choice.
The explanation turned out to be simple: GPT-5 produced only ~20 distinct numeric values out of 100 possible. At that "resolution," 67% of comparisons ended in a tie. Next, 7,000 pairs were run through the pairwise format — and recovery reached 61%. Same judge, same task — just a different question format. The finishing touch: a check across 5 different models (GPT-5.2, Claude Sonnet 4, GPT-4.1-mini, Gemini-2.5-flash, Llama-70B) confirmed the pattern — for all of them, global correlation was higher than within-prompt correlation.
Adaptations
A tournament for picking the best of many
💡 Adaptation for 3+ candidates: when there are more than two, compare them sequentially in pairs — like a tennis bracket.
Round 1:
"Which is better — A or B?" → winner: B
Round 2:
"Which is better — B or C?" → ...
Final:
"Winner of round 1 vs winner of round 2"
Add confidence as a filter
🔧 Technique: confidence → a signal of real difference
If you ask "rate your confidence 1–5" after the verdict, then: - 4–5 → the candidates really differ, you can trust the verdict - 1–2 → the candidates are nearly equal, decide yourself or look for another criterion
This matters most when choosing between two strong candidates — the judge will honestly say "I'm not sure" instead of handing you a random verdict.
Multi-criteria version
💡 Adaptation for complex decisions: compare on one criterion at a time, then tally the wins.
Compare A and B on "clarity" → verdict
Compare A and B on "persuasiveness" → verdict
Compare A and B on "length" → verdict
Result: A won 2 out of 3 → go with A
Works when there's no single main criterion and different aspects matter in different ways.
Resources
Paper: "When LLM Judge Scores Look Good but Best-of-N Decisions Fail"
Author: Eddie Landesberg
Data: Chatbot Arena prompts, via the upstream evaluation pipeline (MT-Bench/Arena line)
Related work mentioned in the paper: Zheng et al. (MT-Bench/Chatbot Arena), Frick et al. (PPE release, PPE-MATH)