Apply to my task
arXiv:2608.25869 ★ 91 Aug 26, 2026

Anchoring Bias in LLM Judges: Mentioning a Past Score Spoils the New Review

Key result

Finding: the LLM judge doesn't care whether the past score was 1.0 or 3.9. All that matters is that it was mentioned at all. That means you can't trust LLM ratings on repeat reviews: 7 out of 8 models systematically lower the new score if they see the old rating's history, even when the text got better. The mere mention of a past score switches the model into "be stricter" mode: GPT-4.1's probability of picking the top score drops from 93% to 0% as soon as a "previous score" field appears in the prompt.

TL;DR

Ask an LLM to rate a revised text while mentioning the score the previous version got, and the new rating gets pulled toward that old number, even if it's random and has nothing to do with the text's actual quality. The model doesn't judge objectively; it latches onto a hint from the context like an anchor.

The researchers proved this the hard way: they deliberately slipped models a random low score as the "previous rating" (while the text itself stayed exactly the same). The result: 7 out of 8 models systematically lowered the new rating, even though nobody asked them to. GPT-4.1, for example, rated the same text 5.0 out of 5 with no context and 4.36 when it saw that "the previous attempt received a low score." At the level of "accept/reject" decisions, this translated into a 22 percentage point drop in the share of accepted texts for one of the models. And it's not just about numbers: on real business data with human labels, this bias blocked error corrections in 48% of cases and flipped a correct verdict into an incorrect one in 10% of cases.

Worse, the standard cures don't work. Asking it to "think step by step" (Chain-of-Thought) doesn't remove the effect. A direct warning to "ignore the previous score" doesn't rescue numerical scores either; it helps a little only for categorical decisions (like "correct/incorrect"). The only reliable fix is to not give the model any context about the past score if you need an independent review.

What's Going On

BAD: The prompt contains "This is attempt #3. Previous score: 2.5"
  → the LLM sees the hint → pulls the new rating toward the old number
  → even if the text got noticeably better

GOOD: The prompt contains no scoring history
  → the LLM rates the text "from scratch"
  → the rating reflects the actual current quality

An important detail: the effect is threshold-based, not proportional. It doesn't matter whether the "past score" was 1.0 or 3.9; what matters is that metadata about a past score is present. Its mere presence sharply shifts the probabilities toward a lower score, while the specific anchor value barely affects the size of the shift.

Example in Action

Task: A copywriter rewrites ad copy for Reels/Shorts and asks ChatGPT to rate its quality on a 1–10 scale after the edits.

A prompt that creates the problem:

Here is the new version of the ad copy. This is the third attempt.
The previous version scored 4 out of 10.

Text: {text}

Rate it on these criteria: is the headline catchy, is there a call to action, 
is the offer clear. Give a score from 1 to 10.

A prompt without the bias:

Rate the ad copy on these criteria: is the headline catchy, is there a call 
to action, is the offer clear. Give a score from 1 to 10.

Text: {text}

Result: In the first case, even if the new text is objectively stronger, the model will most likely give a score closer to the old "4", because it sees the label "this is a rework, the last one was bad." In the second case, the rating will reflect the text's actual quality, untied from its history.

Why It Works

LLMs were trained on texts where people write exactly this way: they treat a past score as a meaningful signal and don't move far enough away from it, even when it's irrelevant. That's the classic human anchoring effect, which the model reproduces functionally without actually "understanding" that the anchor is random.

Token analysis on GPT-4.1 pinned down the mechanics more precisely: without metadata about a past score, the model chose the digit "5" with 93% probability and "4" with 7%. As soon as a "past score" field appeared in the prompt, the probabilities flipped: 100% on "4" and 0% on "5". That's not fine-tuning to a specific number, it's a mode switch: "aha, this is a retry, so I'd better be on guard."

Control knobs: - Remove any fields like "attempt #", "previous score" or "this is a revision" from the prompt → you get an independent rating. - If you need to track progress between versions, ask the model to compare the versions directly ("what changed between version 1 and version 2") rather than give a new score with one eye on the old one. - Don't count on "think step by step" as a cure; the effect lives not in the reasoning logic but in how the context itself is perceived. - An explicit "disregard the previous score" isn't useless, but it works only for "right/wrong" decisions, not for numerical scores.

Prompt Template

The universal principle isn't a technique with steps, it's a prompt hygiene rule:

Rate {item being evaluated} on these criteria: {criteria}.
Give a score from {min} to {max} and a short explanation.

{item being evaluated}: {content}

What to remove from the prompt if you want an unbiased rating: - The attempt / version number - Any mention that this is a "rework" or "revision" - Any past score, even as "context for your information"

If you need to track progress without bias:

Here are two versions of {item}. Don't give an overall score. 
Compare them directly: what got better, what got worse, what stayed the same.

Version 1: {text_1}
Version 2: {text_2}

🚀 Quick start: paste this into the chat:

I regularly ask you to rate revised versions of my text/code/resume 
and mention the past score for context. Help me rewrite my rating prompt 
so that your new rating doesn't depend on what you know about the past rating.
My task: {describe the task}.

The model will strip the scoring-history fields from the prompt and offer either a "clean" rating from scratch or a direct version-comparison format with no scores.

Original from the Paper

Here is the exact metadata bait the researchers added to the prompt to trigger the effect (avoid this in your own rating prompts):

Attempt metadata:
- Attempt: k (This is a revision)
- Prior score: a

Context: The researchers inserted this block into the LLM judge's prompt along with the task, the response and the rubric. k is the attempt number, a is a random number below the acceptance threshold (0–3.99 out of 5). The response text didn't change between conditions; the only variable was whether this block was present.

Limitations

⚠️ Not universal across models: one of the eight models (Llama-3.1-8B) showed almost no effect. A large model is no guarantee of protection, nor a guarantee of vulnerability: the pattern is unpredictable even within one model family.

⚠️ Standard cures don't help: asking for step-by-step reasoning (Chain-of-Thought) doesn't reduce the bias. A direct warning to "disregard the past score" barely helps for numerical scores; it works only for categorical decisions like "right/wrong."

⚠️ Depends on the task type: code review is less susceptible to the effect than creative writing, summarization and factual answers. If you're asking for a code rating, the risk is lower; if you're asking for a text rating, it's higher.

⚠️ Tested on a limited set: the experiment was built on 20 fixed texts and artificially lowered "past scores." In real multi-step revision chains, the size of the bias may differ from what was measured.

How They Tested It

The team took eight models, from GPT-4.1 and Claude to open Llama and Qwen, and ran 192,000 rating requests through them across three variants of the same prompt: no history at all, a hint that "this is a rework," and a full metadata block (attempt number + random past score). The texts being rated (20 fixed tasks: summarization, code review, creative writing, factual questions) didn't change between conditions, only the context around them.

To separate the real effect from the noise of repeated requests, statistics were computed not per individual response but per task, via bootstrap resampling of the 20 tasks 10,000 times. That's the honest approach: if they'd counted all 185,000 valid responses as independent data points, the result would have looked far more certain than it really is.

The most intriguing part is the token analysis on GPT-4.1: the model's probability of choosing "5" over "4" flipped almost instantly and completely once past-score metadata appeared, while changing the anchor value itself within the "bad" range (1.0 vs. 3.9) changed almost nothing. That suggests the effect isn't a subtle mathematical adjustment but a crude switch into "suspicion mode" when the model sees the flag "this is a revision with a low past score."

They also checked it separately on real business data with human labels (not artificial texts), and the bias held up there on categorical decisions: the model blocked corrections of its own errors in 48% of cases and flipped a correct answer into an incorrect one in 10% of cases when it saw a label saying "the past attempt received such-and-such a tag."

Resources

Kapetanovic, A., Altwlkany, K., Mercep, A., Duricic, T., Lacic, E.: Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence, CIKM 2026, Infobip. Code, prompts and statistical formulas: github.com/infobip/llm-judge-bias

Research digest

Key result

Finding: the LLM judge doesn't care whether the past score was 1.0 or 3.9. All that matters is that it was mentioned at all. That means you can't trust LLM ratings on repeat reviews: 7 out of 8 models systematically lower the new score if they see the old rating's history, even when the text got better. The mere mention of a past score switches the model into "be stricter" mode: GPT-4.1's probability of picking the top score drops from 93% to 0% as soon as a "previous score" field appears in the prompt.

Principle

It works not as a gradual shift but as a mode switch. With no scoring history, the model chooses between scores of 5 and 4 at a ratio of 93 to 7. Add the phrase "previous score: 2.5" and the ratio becomes 0 to 100, and it doesn't matter at all whether the anchor was 1.0 or 3.9. The model doesn't compare numbers; it reacts to the very fact that "this is a retry". The wild part is that the anchor's size and value barely affect the outcome; what operates is a binary trigger: "history present / no history."

Why it works

The LLM was trained on texts where people almost always factor someone else's past rating into their new one: the classic human anchoring effect. The model reproduces this pattern literally, without even realizing the anchor is random. On real business data with human labels, this bias blocks error corrections in 48% of cases and flips a correct verdict into an incorrect one in 10% of cases. Chain-of-Thought and asking it to "disregard the past score" barely remove the effect; the anchor sits not in the reasoning logic but in how the context itself is perceived.

When to apply

LLM-as-judge (the model as a judge) → any system with revisions and repeat ratings: code-review agents, essay grading, content moderation, iterative copywriting. Especially important where an honest read on progress between versions matters. Not critical for a one-off rating with no history; there's simply no anchor to latch onto.

Mini recipe

1. Check the prompt: does it contain "attempt #", "version" or "previous score"? Remove all of it if you need an honest, independent rating. 2. To track progress, don't ask for an overall score; ask for a direct comparison of the two versions: what got better, what got worse, what stayed the same. 3. Don't rely on "think step by step": it doesn't cure the anchoring effect; only the anchor's complete absence from the prompt does. 4. If you really need the history, add an explicit "disregard the previous score," but that only rescues right/wrong decisions, not numerical scores.

Examples

[BAD]: `This is the third attempt. Previous score: 4 out of 10. Rate the new text: {text}` [GOOD]: `Rate the text on these criteria: headline, call to action, offer. Give a score from 1 to 10: {text}`

In plain words

When you use a model as a judge, forget about objectivity. Models suffer from a classic human bias: the anchoring effect (anchoring bias). Show an LLM a text's past score and it stops judging on the merits and starts fitting the result to the old number. The model was trained on human texts, and humans always unconsciously cling to the first reference point they see, even if it's a completely random number.

It's like calling an appraiser to a house after a high-end renovation, but first blurting out: "the previous owner sold it for peanuts." The appraiser sees the marble floors but names a laughable price, because his brain is locked onto the peanuts. Same deal with LLMs: a past number in the prompt works like mental hypnosis and completely kills the independence of the rating.

In LLM-as-a-Judge pipelines, any number mentioned earlier breaks the objective scale. Say you've reworked a weak ad script that first scored 3 out of 10. Show the model the old score and even a flawless masterpiece will scrape 5 out of 10 at best, because the model treats the past score as a meaningful signal. Remove that anchor and the same text honestly earns its 9 out of 10.

The study looked at text ratings, but the trap is universal. It breaks prompt A/B tests, automated code review, resume scoring and any task with iterative improvement. Feed the LLM a history of previous attempts with their scores, and all your analytics go down the drain, turning into a self-fulfilling prophecy.

Bottom line: if you want a sound verdict from AI, enforce strict blind testing. Never hand the judge model the history of previous attempts, old scores or "the last reviewer's opinion". A clean context with no anchors is the only condition under which an LLM gives a real quality rating rather than a polite fit to the old answer.

AI lab assistant

Apply this study to your task

Describe what you're working on. AI takes the method and the measured results from this paper and adapts them to your case, or writes a ready prompt built on them.

Based on Anchoring Bias in LLM Judges: Mentioning a Past Score Spoils the New Review arXiv:2608.25869

Sign in to apply this study New accounts get 50 free credits.

Related techniques

LLM-as-a-Judge and Rubric GradingLLM-as-a-judge bias and inconsistent scores problem