LLM judge version drift: a newer model with the same prompt doesn't necessarily judge better

STEP 1: Build a gold set (50–200 examples with human scores)
STEP 2: Freeze the prompt, the rubric, temperature 0. Change ONLY the model
STEP 3: Run the set through the old and the new version → two score tables
STEP 4: Compare per example: old correct + new incorrect = regression
STEP 5: Look at the reasons for regressions (in the model's responses) → decide: fix the prompt or keep the old model
STEP 6: Same thing in reverse, if you want to “cut costs” and roll back to the old model

Steps 3–4 are separate runs. Step 5 can be done with a single request to the LLM itself (template below).

Task: You run content at an online course company. An LLM judge in your automation (n8n or Make, no code) grades the support chatbot's answers on a 0–3 scale: “didn't help / partly helped / helped / perfect.” The gold standard is scores from three of your course coordinators on 80 conversations. A new model version comes out, it's cheaper and “smarter,” and a colleague suggests just changing the model name in the settings. You first check what gets lost.

Prompt (regression analysis, paste into a chat along with the table):

You are a quality analyst for LLM judges. I replaced the judge model with a new version and kept the same prompt.

Below is a table. For each conversation: the coordinators' gold score, the old model's score, the new model's score and its brief rationale.

Task:
1. Find regressions: conversations where old score == gold and new score != gold. Compute their share of the conversations the old model judged correctly.
2. Find improvements: old != gold, new == gold. Compute the share.
3. For each regression, determine the cause from the new model's rationale. Pick one:
   a) over-weighted one aspect (e.g., tone of the answer mattered more than the solution);
   b) interprets the criterion too strictly (“helped” only given for perfection);
   c) latched onto keywords instead of meaning;
   d) other cause (describe it).
4. Group the regressions by cause and show 2 examples per group.
5. Suggest edits to the judge prompt for each group. Don't change anything unrelated to the regressions.

Data:
{conversation_table}

Judge prompt (for reference):
{judge_prompt}

Result: The model will output two shares: regressions and improvements. Then comes a list of regressions grouped by cause, with examples. At the end, targeted edits to the judge prompt. You'll see that the update didn't simply make things “better” or “worse.” Some judgments moved in a different direction, and you can see exactly which way.

The weakness. A prompt doesn't live in a vacuum. It's “fitted” to how a specific model version reads instructions. The new version was trained differently: it has different habits, a different level of strictness, a different weighting of “aspects.” So the same words produce different scores. The average metric hides this: wins in some places offset losses in others.

The strength. Comparing two versions on a fixed set of examples is cheap and unambiguous. You don't have to guess whether “the model is better overall.” It's enough to look at which specific cases broke. You can use the LLM itself as the root-cause analyst: it reads the judge's rationales and groups them.

How the method uses this. A per-example breakdown instead of an “on average” view turns a vague “seems worse somehow” into a concrete list of errors and their causes. And those causes can be fixed by editing the prompt.

The levers: - Gold set size. More examples give stable shares. Fewer is OK, but then don't draw conclusions from isolated cases. - Simple or multi-step prompt. A multi-step rubric (questions first, then a score for each) broke more often on minor updates. On major generation changes it cushioned the drop: for one model pair, losses fell from 28% to 11.5%. If you update “in small steps” often, go simple. If you're switching generations, the rubric gives you insurance. - Direction of the check. Test in reverse too. Rolling back to an older, cheaper model loses much more than moving forward: in one case 21.8% versus 6.7%. - Temperature 0. Lock it so the differences come from the model, not from randomness. Repeated runs in the study produced nearly identical results.

This is a regression test for your LLM automation. The judge itself is your production prompt, which you don't change. Below is a template for analyzing the results:

You are a quality analyst for LLM judges. I replaced the model {old_model} with {new_model}; the judge prompt stayed the same.

Data: {table}. Columns: {id}, {human_gold}, {old_score}, {new_score}, {new_rationale}.

1. Regressions: old_score == gold AND new_score != gold. Compute the share of all correct answers from the old model.
2. Improvements: old_score != gold AND new_score == gold. Compute the share.
3. Assign each regression to a cause:
   — over-weighting one aspect,
   — excessive strictness,
   — keyword matching instead of meaning,
   — other (describe).
4. Show 2 examples per cause.
5. Suggest prompt edits for each cause.
6. Give a verdict: is it safe to switch models, does the prompt need work, or is it better to stay on the old one.

Plug in the model names and the table. The gold standard must come from people, not from another model.

🚀 Quick Start — paste into the chat:

Here's a template for regression-testing a change of LLM judge model. Adapt it to my task: {what_you_grade_and_on_what_scale}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about your rating scale, the format of your table and where the gold standard comes from. It needs this to compute regressions correctly. It keeps the template's structure and fits it to your format.

⚠️ You need a human gold standard. Without a hand-labeled set, regressions can't be counted. You'll have to spend some time once to grade a few dozen examples.

⚠️ One domain. The experiments were run on relevance assessment (does a text passage fit the query) on two TREC sets. For other judging tasks the mechanics are probably the same, but the numbers don't carry over.

⚠️ Small open models. For Qwen and Llama, the models used were around 7–9B parameters. Larger models may behave differently. For early Llama versions, the authors themselves note that the rubric judgments were incomplete.

⚠️ Regression causes are described only broadly. Three mechanisms (over-weighting an aspect, strictness, keywords) are named, but there's no universal recipe for fixing them. You'll have to find and test prompt edits yourself.

⚠️ No “just improve the prompt” step. The authors didn't test whether redesigning the prompt restores the lost quality. They show the problem and the need for re-testing.

The idea was simple: freeze everything except the model. The team took four families (Gemini, GPT, Qwen, Llama) and successive versions within each. Two fixed judge prompts: UMBRELA (a single request with a description of the criteria and a step-by-step procedure) and EXAM (rubric questions are generated first, then the passage is scored against them). The task: assign a 0–3 score to “query — text passage” pairs and compare with human scores. To avoid mixing effects, the EXAM rubric was generated once by the oldest model in each family and used for all of them.

First they looked at average metrics: exact agreement, mean error, correlation. No single “newer is better” picture emerged: for Gemini Flash the gains plateaued over time, for Qwen accuracy jumped around. For GPT mini, moving from 4o to 5 dropped exact agreement for the simple prompt from about 56% to 36%, while for the rubric it went up. So the effect of an update depends on the prompt.

Then they went down to individual examples. Even when the average went up, correct judgments were still lost. For example, switching from GPT-4o mini to GPT-5 mini lost 28% of the simple prompt's correct judgments. For Qwen 2.5 → 3 it was 22.6%. Meanwhile, the latest minor Gemini updates (3.7 → 3.8) gave only 2.3%. This suggests that major generation changes are riskier.

There were two surprises. The complex rubric broke more on minor updates, possibly because a single step in the chain changes the outcome. But on big shifts, the rubric acted like “scaffolding” and held quality up. And rolling back to an older model loses more than moving forward. In practice, that means a cheap old model isn't free. For robustness, the authors repeated the Gemini run three times, and the results barely changed. So it's not API noise, it's the models themselves.

🔧 Technique: lock in a “gold set” → one command for any update.

Keep your 50–100 labeled examples alongside the prompt. Whenever the model in your automation is updated, run them first, and only then switch over the production flow. This is the authors' idea turned into a habit instead of a one-off check.

🔧 Extrapolation to agents (an adaptation idea, not tested in the paper):

If you configure an agent through CLAUDE.md or a system prompt, save 10–20 typical tasks with accepted results. After a model change, run them and compare with the previous results. The agent's behavior under the same rules may shift too.

Copy

The repository has a tests/golden_tasks.md file with 15 typical tasks and their expected results.
After a model change: complete each task, compare it with the expected result and list the discrepancies with their causes.

• Paper: The Impact of Backbone Evolution on LLM-Based Relevance Assessments, Chuting Yu, Guido Zuccon, Teerapong Leelanupab, University of Queensland, Australia
• Code and results: https://github.com/ielab/judge-backbone-evolution
• UMBRELA — Upadhyay et al., an open implementation of the Thomas et al. approach
• EXAM — Farzi & Dietz, a question-based rubric
• Dietz et al. — LLM evolution as a threat to the reproducibility of assessments
• Data: TREC Deep Learning 2019 and 2020, passage ranking