Trustworthy Verdict Rate (T): testing an LLM judge for repeatability, answer order and accuracy

STEP 1: Collect 30–50 answer pairs where you yourself know which is better → reference
STEP 2: Run the judge in AB order and in BA order, 3–5 times each → verdict table
STEP 3: Compute three numbers → C, φ, A
         C = share of verdicts that repeat on a re-run
         φ = share of pairs where the verdict changes when A and B are swapped
         A = share of verdicts that match the reference
STEP 4: T = C × (A − φ/2)  → “share of verdicts you can trust”
STEP 5: Diagnosis from two numbers (C and φ) → choose the treatment:
         C high, φ low   → reliable judge
         C low, φ low    → noisy but honest: repeat and take the majority
         C low, φ high   → chaos: both orders + repeats
         C high, φ high  → “confidently biased”: the most dangerous
STEP 6: Change the evaluation format: a scored rubric in ONE request (G-Eval) instead of a pairwise “A or B”

The accuracy ceiling is simple: A ≤ 1 − φ/2. If the verdict flips in 60% of pairs, accuracy can't exceed 70%, no matter how smart the model is. Until you bring φ down, a smarter model won't help.

Task: You run support at a bank like Chase. A bot answers customers, and you're testing two versions of its prompt. Reading 500 answer pairs by hand is impossible, so Claude or GPT will be the judge. But you want to know whether you can trust it before making a decision.

Judge prompt (scored rubric in a single request):

You are evaluating two answers from a bank's support bot to the same customer request.

Customer request: “Why was my $5,000 transfer blocked and what should I do?”

Bank policy: transfers over $3,000 to new recipients go through
an additional review of up to 24 hours. The customer can confirm the transfer
in the app or call the number on the back of their card. Internal
anti-fraud rules must not be disclosed.

Answer 1: {answer_1}

Answer 2: {answer_2}

Score EACH answer on four criteria from 1 to 5:
1. Accuracy: doesn't contradict the policy, no made-up facts
2. Completeness: the reason is explained and all available actions are named
3. Clarity of next step: the customer understands what to do right now
4. Tone: polite, no corporate-speak and no excuses

First output a score table, then one line:
VERDICT: Answer 1 or Answer 2.

How to run it: run the prompt on 40 pairs where you know the correct verdict. Run each pair twice: first as is, then with “Answer 1” and “Answer 2” swapped. Repeat each variant 3 times. Count how many pairs had their verdict flip.

Result: For each run, the model outputs a score table and a “VERDICT” line. You get three numbers: how well verdicts repeat, how often they change when swapped, and how often they match your reference. If the verdict flips often, the judge can't be trusted in this form. Then accept a verdict only when it matches in both orders. Send disputed pairs to manual review.

The model's weakness. A model comparing two texts is sensitive to order. The first or second answer gets an advantage simply because of its position. This isn't a reasoning error; it's a skew toward a particular slot. At temperature 0 the skew looks like “stability”: the same thing every time, because every time it's “I pick the first one”. If you only look at repeatability, the problem stays invisible.

The model's strength. When a model scores a single answer on concrete criteria, it works more reliably. Criterion scores depend less on what's sitting next to the answer. If you ask for all criteria in one request, the model keeps them together and repeats itself better. When the evaluation is chopped into four separate calls, each call gets a slightly different context. The totals diverge.

How the method uses this. T breaks “reliability” into independent parts, and each one has its own fix. Noise across re-runs is fixed with repeats and majority voting. Order bias is fixed with both orders. Weak evaluation is fixed by changing the format or using a stronger model. Without the breakdown, you treat the wrong ailment.

Control knobs: - Number of pairs in the sample — fewer pairs is cheaper but widens the spread of the estimate. For a first check, 30–50 is enough. - Number of repeats — 3 give a rough picture, 5–10 a more precise one. - Evaluation format — a scored rubric in one request vs. a bare “A or B”. The study showed format can shift T more than switching models. - Output order — “verdict first, then justification” or the reverse. The effect depends on the model; test on your own data. - Criteria — 4 criteria in the study. Pick your own criteria to fit the task.

You are a judge. Compare two answers to the same task.

<Task>
{task_description_and_context}
</Task>

<Rules_and_reference>
{policy_or_quality_criteria}
</Rules_and_reference>

<Answer_1>
{text_1}
</Answer_1>

<Answer_2>
{text_2}
</Answer_2>

Score EACH answer on the criteria from 1 to 5:
1. {criterion_1}
2. {criterion_2}
3. {criterion_3}
4. {criterion_4}

Output a score table. Then one line:
VERDICT: Answer 1 or Answer 2.
No ties allowed — pick one.

Run procedure (manually or via no-code automation):

1. Run the template on each pair as is (AB order).
2. Run it again with {text_1} and {text_2} swapped (BA order).
3. Repeat both runs 3–5 times.
4. If the verdict changes when swapped — the pair is “disputed”.
5. Compute: C (repeatability), φ (share of disputed pairs), A (match with reference).
6. T = C × (A − φ/2).

Fill in: - {task_description_and_context} — what was being answered and to which request. - {policy_or_quality_criteria} — the internal rules the judge should rely on. - {criterion_N} — 3–4 concrete, measurable criteria. “Creativity” is bad; “the answer names the customer's next action” is good. - {text_1} and {text_2} — the two variants to compare.

🚀 Quick start — paste into the chat:

Here's an LLM judge template with a rubric, plus a procedure for checking its reliability.
Adapt it to my task: {what I'm comparing and why}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what you're comparing, which criteria to score on, and what a correct answer looks like. The criteria matter because the method rests on concrete scores. Vague criteria will drop you right back to a bare “which is better”.

⚠️ There's no universally best format or judge: the ranking of models and formats changes from dataset to dataset. The “judge × task” interaction turned out to be the main source of disagreement. Test on your own data; don't take someone else's conclusion as a given.

⚠️ A single-request rubric doesn't help everyone: it gave strong models a big boost and weak ones much less. Don't carry a single result on one model over to another without testing.

⚠️ Splitting evaluation into several requests hurts: the variant where each criterion was scored in a separate call badly hurt repeatability. For most models, the final reliability ended up lower than with a plain direct comparison.

⚠️ You need data with known answers: without reference pairs you can compute only C and φ, not accuracy. For that you need a human to label 30–50 pairs.

⚠️ Two-way choice only: the method is designed for a forced choice between A and B. Ties, scale ratings and “neither fits” aren't built into the formula. The paper in the provided excerpt cuts off at the conclusions, so the authors' full list of recommendations isn't visible.

⚠️ Temperature can't always be changed: for reasoning models, temperature isn't available via the API. Repeats for them are noisy in their own way.

A team from PayPal took six judge models: GPT-5.2, GPT-4o, Gemini 2.5 Pro, Gemini 2.0 Flash, Claude Sonnet 4.6 and Llama 3.3 70B. Each was tested on four datasets: two about subjective preferences (MT-Bench, AlpacaFarm) and two with an objective answer (JudgeBench, LLMBar). They varied everything at once: five output formats, two answer orders, three temperatures and ten repeats of each condition. They compared this with common practice: looking only at accuracy and repeatability.

The results were unpleasant. Even at temperature 0, 7% to 17% of verdicts changed across identical runs. When the answers were swapped, the verdict flipped for almost every pair. The most telling case: Gemini 2.0 Flash on JudgeBench with a “verdict first” format in XML. Repeatability 100%, flip rate on swap 98%, accuracy 51%. T in that case is just 2%. The judge with perfect repeatability turned out to be a coin flip.

Output format mattered more than expected. For Gemini 2.5 Pro, switching from a “single token” to “verdict + justification” in JSON raised T from 29% to 77%. Meanwhile the flip rate fell from 56% to 7%. Temperature, on the other hand, only affected repeatability: it had almost no effect on “accuracy minus bias”.

Comparing three approaches gave the main practical lesson. A rubric in a single request (G-Eval) raised T for Gemini 2.5 Pro from 29% to 78%. Splitting the evaluation into four requests (FLASK) tanked repeatability, for example from 86% to 35% for GPT-5.2.

The practical takeaway: repeatability is easy to mistake for reliability, and checking answer order costs almost nothing.

🔧 Technique: accept a verdict only when it matches in both orders → fewer false confident decisions

The paper recommends averaging over both orders. A simple practical version: run the judge twice (AB and BA). If the verdicts match — accept. If not — flag the pair as disputed and hand it to a human.

Run the pair in AB order and in BA order.
They match → verdict accepted.
They don't match → status “disputed, needs human review”.

This is my adaptation of an idea from the paper, not the authors' protocol.

🔧 Technique: repeatability as a confidence signal → an early filter for disputed pairs

The authors note that agreement across repeated requests serves as a confidence signal without calibration. Run the judge 3 times on one pair. If the verdicts diverge, the pair is hard. But remember: if order is skewed (high φ), repeatability will be high and deceptive. Check order first, then repeats.

Extrapolation for agents: in a reviewer agent's instructions file (for example, CLAUDE.md) you can write: “When comparing two solutions, evaluate them twice: first in the order A, B, then B, A. If the conclusions don't match, don't make a choice — describe how the solutions differ.” This is my idea based on the paper's results; the paper itself didn't test it for agents.

Paper: All Verdicts are Not Equal: Rethinking LLM Judge Reliability

Authors: Vineet Kumar, Darshita Rathore, Anindya Moitra — PayPal Artificial Intelligence, Bangalore, India.

Methods and datasets mentioned: G-Eval (Liu et al., 2023), FLASK (Ye et al., 2024), MT-Bench (Zheng et al., 2023), AlpacaFarm (Dubois et al., 2023), JudgeBench (Tan et al., 2025), LLMBar (Zeng et al., 2024).