Repeat Your LLM-Judge Scores: A Single Model Score Is Noise, and the “Improvement” After Edits Depends on Who's Judging

STEP 1: Score the material N times, each time in a NEW chat (independent runs) → N scores
STEP 2: Average them (or take the median) → estimate of the judge's “expected score”
STEP 3: Make your edits → repeat steps 1–2 for the new version
STEP 4: Improvement = mean “after” − mean “before”
STEP 5: Repeat steps 1–4 with a DIFFERENT judge model
STEP 6: The improvement is confirmed only if both models see it going the same way

Every run is a separate request. You can't get independent runs inside one chat: the model sees its previous scores and adjusts to them.

Task: The owner of an online school built on Squarespace rewrote the landing page for the course “Excel for Bookkeepers” and wants to know whether it actually got better. Until now, they asked ChatGPT to rate the page once before the edits and once after, and went by the difference.

Prompt (paste into a new chat, 5 times for the original and 5 times for the new version):

You are evaluating a sales landing page for an online course aimed at a US audience (price $149, target audience: bookkeepers aged 30–45, promoted through Facebook/Instagram ads and Google Ads).

Evaluate it on these criteria: clarity of the offer, trust (case studies, testimonials, guarantees), handling of objections, call to action.

Output a final score from 1 to 10. The score must have a non-zero fractional part (for example, 6.4 or 7.8, but not 7.0).

Response format:
1. Assessment for each criterion (1–2 sentences)
2. Final score: X.X

Landing page text:
{landing_page_text}

Protocol: 5 runs on the old version and 5 on the new one, each in a new chat. Then the same 10 runs in another model (for example, ChatGPT and Claude). Compare the averages.

Result: In each run the model gives a breakdown across the four criteria and a fractional final score. Scores will differ slightly from run to run. The averaged values show whether the rating moves in the same direction for both models. If one model shows an improvement and the other doesn't (or shows a drop), the improvement is most likely a matter of taste, not a general one.

The LLM's weakness. A model doesn't “know” the score; it picks a number from a distribution. Even under identical conditions, neighboring runs land on 7 one time and 8 the next. The difference between “before” and “after” carries noise on both sides, so one run per version proves almost nothing. On top of that, every model has its own preferences: two models can rate the original the same way yet react differently to specific edits.

The LLM's strength. Averaging independent runs cuts noise nicely: the more runs, the more precisely you pin down the judge's “average” position. And if both models see an improvement, it's much more likely to be real rather than a matter of taste.

Control levers: - Number of runs → you need more when the differences between versions are small. For a rough “worse/better”, a few will do. For subtle differences, you need a lot. - Scale → a fractional score reduces the spread but shifts the mean (upward, in the study). Only compare within the same scale. - Number of judges → several different models beat one: it's your taste check. - Independence → a new chat for every run. Otherwise the model adapts to its earlier answers.

Role: you are an independent evaluator. Evaluate the material strictly against the criteria below, without comparing it to other versions and without referring to previous scores.

Context: {context_and_audience}

Criteria:
{criterion_1}
{criterion_2}
{criterion_3}

Output a final score from 1 to 10. The score must have a non-zero fractional part (for example, 6.4, but not 7.0).

Response format:
1. Assessment for each criterion (1–2 sentences)
2. Final score: X.X

Material:
{material}

What to fill in: {context_and_audience} is who the material is for and what it's supposed to achieve; {criterion_N} is the 3–5 criteria you're scoring against; {material} is the text of the version. Run the same template several times in new chats, for each version and for each model.

🚀 Quick start: paste this into a chat:

Here is a template for scoring material with a fractional score. Adapt it to my task: {your task: what I'm evaluating and why}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask who the target audience is, which criteria matter and what counts as a good result. That's how the criteria get concrete: vague criteria increase the spread of scores between runs. It'll take the structure from the template and assemble a ready-to-use prompt for your task.

⚠️ There's always noise: even under identical conditions, scores wobble. The swings are mostly one point, but for materials of similar quality that's enough to flip the ranking.

⚠️ A fractional scale shifts the mean: the spread drops, but the average score goes up. You can't compare results from a fractional scale with results from an integer scale.

⚠️ An improvement from one model doesn't prove anything: agreement on originals doesn't guarantee agreement on edits. The exact same edits got a plus from one model and a minus from another.

⚠️ Subtle differences need many runs: quarter-point precision takes dozens of repeats. Half-point precision takes several times fewer, but still more than one.

⚠️ Tested on peer review of scientific papers: that's one genre, and that's what the models judged. Carrying this over to copy, landing pages or résumés makes sense, but the authors didn't test it.

⚠️ An average isn't the truth: the mean of several runs estimates the “model's taste”, not objective quality. The models' correlation with human ratings is moderate.

The researchers took 200 papers from the ICLR conference (100 each from 2025 and 2026), selected so that a range of topics and a range of average human scores were represented. For each paper they requested ten independent reviews from each of three model configurations, 6,000 scores in total. Then, adding experiments with the scale and with edits, they brought the total number of reviews to roughly 14,000.

First finding: for most papers the scores barely diverge (within one point), but fully identical sets of ten scores are not that common, from a third to half of the papers. For ranking, this means that with a single run the order of two papers reproduces in about 60% of cases, and with five averaged runs, in 75–85%. Based on the authors' calculations, estimating the expected score to within half a point with high probability takes roughly 6 to 12 runs, and a quarter point takes up to 45 (with the worst-case guaranteed bound being several times higher).

Second experiment: forcing the model to output fractional scores instead of whole numbers. The spread dropped by 13–16%, but the average scores rose (by 0.27 and 0.69 points depending on the model). So the scale changes not just precision but position too.

The most interesting part is the third experiment. One model wrote edits for the papers, another (or the same one) decided which to accept, and then two models scored the resulting versions. Luna rated the exact same edits at +0.47 on average, and Haiku at −0.35. Swapping the model that picked the edits barely mattered: the judge is what decides. Meanwhile, on the original papers the two judges' scores correlated highly (0.80), but on the improvements the correlation fell to 0.4 and even 0.09. Part of the gap is explained by noise, but even after correcting for noise, agreement on improvements (around 0.5) stays noticeably lower than on originals (0.84).

The practical insight: models agreeing on an “as is” score doesn't mean they agree on “what got better”. That's why edits need to be confirmed by multiple judges.

🔧 Technique: drop the fractional scale → compare in whole points only

If you need to compare against past scores, stick with whole points. In the template, replace the line about the fractional part with “The score is a whole number from 1 to 10.” The spread will be a bit larger, but the scale won't drift relative to your old data.

🔧 Technique: add a second judge to your prompt loop

For an “edit → score” loop in chat, run the scoring in two different models. Accept an edit only if both show an improvement in the same direction.

Extrapolation (my idea, not tested in the paper): a rule for agents. In the instruction file of an agent (CLAUDE.md, AGENTS.md) that iteratively improves text based on LLM scores, you can add:

Copy

Rule for evaluating edits:
- Don't treat an edit as an improvement based on a single model score.
- For each version, request a score at least {N} times independently and take the mean.
- Accept an edit only if the improvement is confirmed by a second judge model.
- If the two judges disagree on the sign of the improvement, roll back the edit and report it.

This carries the principle of “don't trust one score or one judge” over to agentic self-improvement loops.

• Paper title: Can We Believe AI Peer Review?
• Authors: Zongrui Yang, Haoyuan Li, Zhongsheng Wang, Gillian Dobbie, Jiamou Liu
• University: University of Auckland, Auckland, New Zealand
• References from the study: ReviewEval (Garg et al., 2025), TitleTrap (Du, 2025), LLM-as-a-Reviewer (Li et al., 2026a), Rhetoric (Li et al., 2026b), G-Eval (Liu et al., 2023), Stureborg et al., 2024, Chiang and Lee, 2023