Apply to my task
arXiv:2603.12520 ★ 91 Mar 12, 2026

LLM-Judge Scores Lie: Pairwise Comparison Nearly Tripled the Value of Head-to-Head Picks

Key result

67% of comparisons end in a tie when you ask an LLM to score several candidates — and the choice becomes a lottery. The real value of such a judge: 21% of the possible maximum. In a best-of-two test with the same GPT-5 judge, pairwise comparison cut ties from 60% to 4% and raised the value to 61%. The method helps pick the better of two options without random ties (tested on chatbot answers; headlines, offers and support replies are our extrapolation). With four candidates it did not always win. The trick: not "score from 0 to 100" but "which is better — A or B?" The numeric scale yields only ~20 real levels out of 100 — two solid candidates land in the same bucket and get the same score. Pairwise comparison changes the task. In 97.9% of cases where the numbers gave a tie (and the reference preferred one answer), the pair found a winner — and matched the reference in 78% of them.

01

TL;DR

When you ask an LLM to score several candidate answers (0–100) so you can pick the best one, the method works far worse than it looks. Instead, ask directly: "Which is better — A or B?" Sounds like a small thing, but the difference is huge.

The root of the problem: numeric scores produce too few distinct values (~20 levels out of 100), so 67% of comparisons end in a tie — the model gives both candidates the same score, and the choice becomes random. The global correlation looks decent (r = 0.47), but that's an illusion — the metric is propped up by the obvious cases, and when choosing among answers to the same prompt the judge (GPT-5) captures only 21% of what a perfect choice would deliver.

Pairwise comparison ("A, B, or tie?") cuts ties from 60% to 4%, and the value of the choice rises from 21% to 61% (a best-of-2 test with the same GPT-5 judge). It is not a universal win: in a best-of-4 test with GPT-4.1-mini comparing all pairs, pairwise recovered less than numeric scores (12% vs 26%). The model doesn't get smarter — it just stops "abstaining" through identical scores and is forced to make a decision.

Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.

02

Method Blueprint

OLD WAY (broken):
  Request → score A and B on a 0-100 scale 
  → A: 72, B: 72 → tie → random pick
  → Value: 21%

NEW WAY (works):
  Request → which is better: A or B?
  → explicit choice + reason
  → Value: 61%

If there are more than two candidates → pairs in turn (A vs B, winner vs C...)
  (our suggestion; the 21% → 61% result is for choosing between two)
03

Example

The task and prompt below are our illustration, not from the paper.

Task: You've written three versions of the pitch for your real-estate newsletter on Substack and want to keep the best one.

Prompt (pairwise comparison):

Evaluate two descriptions of a real-estate newsletter. Decide 
which one is better at hooking a potential subscriber.

Description A:
"A newsletter about the Chicago real estate market. Analysis, trends, tips 
for buyers and investors."

Description B:
"I help you avoid losing money on real estate: I break down 
specific properties, explain the risks and calculate the real return."

Which is better for a subscriber — A, B, or are they equal?
First explain your reasoning, then give a clear verdict: A / B / TIE.

Expected result: The model first shows its reasoning (why one is more specific, why the other is vaguer), then delivers an unambiguous verdict. If you have three versions, compare the first two first, then pit the winner against the third.

04

Why It Works

The problem with numeric scores. When you ask for "a score from 0 to 100," the model thinks in coarse buckets: "good" → 70–80, "very good" → 85–90. When comparing two solid candidates, both land in the same bucket, get the same score — and the choice becomes a coin toss. That's not the model being lazy; it's a structural problem with the numeric format.

Why pairwise comparison works. When you ask "A or B?", the model can't escape into a tie through identical numbers. It's forced to find at least some difference. And, importantly, it does see the difference — the numeric scale just didn't let it express it. In the paper, among pairs where numeric scores tied but the reference labels preferred one answer, pairwise comparison named a winner 97.9% of the time and matched the reference 78.5% of the time.

Control knobs: - "A / B / TIE" — keep TIE so the model doesn't invent a winner where things really are equal - "explain your reasoning" — put it before the verdict so you can see the logic and argue with it - Confidence — after the verdict, you can add "rate your confidence from 1 to 5" (the paper's pairwise prompt did this) → treat it as a rough hint only: the paper found the judge poorly calibrated about its own errors - Many candidates → our suggestion, not tested in the paper: run a knockout tournament instead of every pair (the winner of pair 1 faces the winner of pair 2)

05

Prompt Template

Evaluate two versions of {what_we_compare}.

Version A:
{version_A}

Version B:
{version_B}

Selection criterion: {criterion}

First explain how they differ on this criterion.
Then give your verdict: A / B / TIE.
Rate your confidence: 1 (nearly equal) — 5 (clear winner).

What to fill in: - {what_we_compare} → an offer, an email subject line, a product description, a support reply - {criterion} → conversion, clarity, persuasiveness, fit with the brand voice - {version_A} and {version_B} → your full texts


🚀 Quick start — paste into a chat:

I want to compare several versions of {what you have} 
and pick the best one using pairwise comparison. 
Adapt this template to my task and ask me clarifying questions:

[paste the template above]

The LLM will ask what you're comparing, by which criterion, and how many candidates there are — because pairwise comparison needs to be tuned to the specific task and number of candidates.

06

Limitations

⚠️ Only works when there are real differences: If the candidates are nearly identical, even pairwise comparison gives an unreliable result (accuracy drops from 94% when the reference shows a large quality gap to 66% when the gap is small).

⚠️ Not a guaranteed win with more candidates: Comparing every pair of 4 candidates takes 6 calls. In the paper's best-of-4 test with all pairs and GPT-4.1-mini as judge, pairwise recovered less value than numeric scores (12.3% vs 26.2%). The authors: pairwise helps a lot in head-to-head best-of-2, but "does not automatically win" in best-of-4.

⚠️ Calibration won't fix the quality metrics: The author checked — "adjusting" numeric scores through calibration barely helps: plain calibration left recovery unchanged, and even adding response length raised it by only 2.6 percentage points. The problem is the judge's ranking within a prompt, not the scale.

⚠️ Doesn't replace objective evaluation: If there's a correct answer (math, code), check it directly rather than through a judge.

07

How It Was Studied

The researcher took 5,000 prompts from Chatbot Arena — real user questions — and took 4 candidate answers for each, generated by Llama-3.3-70B and Llama-405B setups. GPT-5 was appointed judge and asked to score each answer on a 0 to 100 scale. Then the question was: how often does "best by score" match the best answer according to the dataset's reference labels?

The result was surprising. The global correlation between the judge's scores and real quality looked tolerable (r = 0.47), but when they measured what actually matters — how much the judge improved the choice compared to random — the number was dismal: 21%. In other words, the judge delivered only a fifth of the value of a perfect choice.

The explanation turned out to be simple: GPT-5 produced only ~20 distinct numeric values out of 100 possible. At that "resolution," 67% of comparisons ended in a tie. Next, 7,000 pairs were run through the pairwise format — and in this best-of-2 setting recovery reached 61% (vs 21% for scores on the same pairs). Same judge, same task — just a different question format. The finishing touch: a check across 5 different models (GPT-5.2, Claude Sonnet 4, GPT-4.1-mini, Gemini-2.5-flash, Llama-70B) confirmed the pattern — for all of them, global correlation was higher than within-prompt correlation.

08

Adaptations

A tournament for picking the best of many

💡 Adaptation for 3+ candidates (our extrapolation, not tested in the paper): when there are more than two, compare them sequentially in pairs — like a tennis bracket.

Round 1:
"Which is better — A or B?" → winner: B

Round 2:
"Which is better — B or C?" → ...

Final:
"Winner of round 1 vs winner of round 2"

Add confidence as a filter

🔧 Technique: confidence → a signal of real difference

If you ask "rate your confidence 1–5" after the verdict, a possible rule of thumb (ours): - 4–5 → likely a real difference - 1–2 → the candidates may be close, decide yourself or look for another criterion

Use it as a rough hint, not a guarantee: the paper found the judge poorly calibrated about its own errors, and recommends asking for explicit probabilities rather than relying on token probabilities.

Multi-criteria version

💡 Adaptation for complex decisions (our extrapolation, not tested in the paper): compare on one criterion at a time, then tally the wins.

Compare A and B on "clarity" → verdict
Compare A and B on "persuasiveness" → verdict
Compare A and B on "length" → verdict

Result: A won 2 out of 3 → go with A

Works when there's no single main criterion and different aspects matter in different ways.

09

Resources

Paper: "When LLM Judge Scores Look Good but Best-of-N Decisions Fail"

Author: Eddie Landesberg

Data: Chatbot Arena prompts, via the upstream evaluation pipeline (MT-Bench/Arena line)

Related work mentioned in the paper: Zheng et al. (MT-Bench/Chatbot Arena), Frick et al. (PPE release, PPE-MATH)

Research digest

Key result

67% of comparisons end in a tie when you ask an LLM to score several candidates — and the choice becomes a lottery. The real value of such a judge: 21% of the possible maximum. In a best-of-two test with the same GPT-5 judge, pairwise comparison cut ties from 60% to 4% and raised the value to 61%. The method helps pick the better of two options without random ties (tested on chatbot answers; headlines, offers and support replies are our extrapolation). With four candidates it did not always win. The trick: not "score from 0 to 100" but "which is better — A or B?" The numeric scale yields only ~20 real levels out of 100 — two solid candidates land in the same bucket and get the same score. Pairwise comparison changes the task. In 97.9% of cases where the numbers gave a tie (and the reference preferred one answer), the pair found a winner — and matched the reference in 78% of them.

Principle

Numeric scoring and pairwise comparison are different tasks for the model. "Score from 0 to 100" forces it to think in buckets: "good" → 75, "very good" → 85. Two solid candidates land in the same bucket — result: 75 vs 75, a coin toss. "A or B?" removes the option of escaping into a tie through identical numbers — the model is forced to find at least some difference. And it does see it — the numeric scale just gave it no tool to express the difference between 72 and 73. Add "explain your reasoning before the verdict" — you'll see the logic and can argue with it. Add "rate your confidence from 1 to 5" — a low score hints the candidates may be close, but treat it as a rough signal: the paper found the judge poorly calibrated about its own errors.

Why it works

The problem isn't that the model is lazy. The problem is the format. In practice, the 0–100 range yields ~20 distinct values — the model thinks in big blocks, not precise numbers. Twenty levels across a hundred possible answers means mathematically guaranteed collisions. A global correlation of r=0.47 looks decent, but it's misleading: the metric is inflated by obvious cases — when a bad candidate is compared with an excellent one. Within one prompt, where you have to choose between solid options, the judge does only a little better than guessing (21% of the possible gain). The author tested "calibrating" the numeric scores — it didn't help (at most +2.6 percentage points of recovery). It's not about the numbers but about the task format itself.

When to apply

Picking the best of several options: email subject lines, offers, product descriptions, support replies, article drafts — especially when the candidates are close in quality and you need a reasoned choice, not a coin toss. For 4+ candidates — our suggestion is a tournament: the winner of the first pair faces the winner of the second. Not tested in the paper; its all-pairs best-of-4 test with a smaller judge did worse than scores. NOT a fit for: objectively checkable tasks (math, spelling, code) — check those directly; near-identical candidates — accuracy drops from 94% on large quality gaps to 66% on small ones.

Mini recipe

1. Give two concrete candidates: paste the full texts, don't paraphrase. The model has to see exactly what the user will read. 2. Name one criterion: conversion, clarity, fit with the tone, persuasiveness. One — not three. 3. Ask for the reasoning before the verdict: that way you see the logic, not just the answer. You can argue with it or use it as an insight. 4. Demand a clear verdict: A / B / TIE. Keep TIE — otherwise the model will invent a winner where things really are equal. 5. Add confidence: "rate from 1 to 5, where 1 is nearly indistinguishable and 5 is a clear winner." Treat a low score as a hint the candidates may be close, not as proof. 6. For three or more candidates — a tournament (our suggestion): first A vs B, then the winner vs C. Don't run every pair at once — that's 6 comparisons for 4 candidates, and you don't always need them.

Examples

[BAD]: `Score three email subject lines on a scale from 0 to 100 and tell me which is best` [GOOD]: `Evaluate two email subject lines for small-business owners. Subject A: "5 ways to lower your taxes in 2025" Subject B: "How we saved a client $25,000 in taxes in one quarter" Criterion: which subject line is a stronger motivation to open the email. First explain how they differ on this criterion. Then give your verdict: A / B / TIE. Rate your confidence: 1 (nearly equal) — 5 (clear winner).`

In plain words

When you ask a model to rate several candidate answers on a 0-to-100 scale, you expect Swiss-watch precision and instead get a shot in the dark. The problem is how an LLM's brain is wired: it can't do fine-grained calibration of numbers. To the model, scores are just broad brushstrokes. It knows the difference between "garbage" and "decent," but when it comes to choosing between two strong candidates, it just slaps the same 85-90 points on both and washes its hands. Picking the best ends up being a coin flip, even though you sincerely trust the numbers.

It's like walking into a restaurant and asking the waiter to rate the steak and the fish on a 100-point scale. He'll say both are a 90 because the chef is on fire tonight. But if you ask, "What should I order right now?", he'll instantly say: "Get the steak, it's perfect today." In the first case he's trying to be an objective judge and loses the nuance; in the second he makes a concrete choice based on a direct comparison. LLMs work exactly the same way: numbers confuse them, while a direct question makes them focus on the details.

The research is clear: Best-of-N (picking the best from a batch) via scoring works much worse than it looks — the GPT-5 judge captured only 21% of the possible gain. To squeeze the most out of the model, use pairwise comparison. Instead of torturing the model with a scale, give it two options and ask: "Which is better — A or B?" In this mode the model notices microscopic differences in style, logic or delivery that get completely erased when it tries to cram its impressions into abstract numbers. In a head-to-head test (best of two), the value of the pick rose from 21% to 61%.

It was tested on chatbot answers to real user questions, not on ads or headlines, so applying it to picking a value proposition, a headline or a support reply is our extrapolation — but a reasonable one: when two options are close, a direct question beats a rating. And it's not magic: with four candidates and a smaller judge, comparing all pairs actually did worse than scores. Turning subjective quality into an "objective" 8.5 out of 10 often hides ties that quietly make your choice random.

Bottom line: stop playing mathematician and asking the model for scores. When you need to pick between two close options, make the model compare them head-to-head. In that setting direct comparison beat scores by a wide margin. Anyone who blindly trusts a GPT "average score" risks getting a random result in pretty wrapping.

AI lab assistant

Apply this study to your task

Describe what you're working on. AI takes the method and the measured results from this paper and adapts them to your case, or writes a ready prompt built on them.

Based on LLM-Judge Scores Lie: Pairwise Comparison Nearly Tripled the Value of Head-to-Head Picks arXiv:2603.12520

Sign in to apply this study New accounts get 50 free credits.