EditJudgeBias: How Multimodal AI Judges Get Thrown Off by Things That Have Nothing to Do With Edit Quality

The audit from the paper (for the reader, it's a checklist of "what can throw the judge off"):

STEP 1: Take an edit (source + instruction + result) → judge's score WITHOUT a cue
STEP 2: Add an irrelevant cue that doesn't change quality → judge's score WITH the cue
        (A/B order, "majority opinion," model name, descriptive caption,
         sticker, frame, watermark, brightness, saturation)
STEP 3: Measure the judge's "noise": repeat the same request + an empty intervention (SHAM)
STEP 4: If the shift exceeds the noise → it's judge bias, not edit quality

A defense for practitioners (based on the paper's findings):

STEP 1: Judge prompt: neutral, no outside opinions or model names
STEP 2: Run 1: order A, B  → verdict
STEP 3: Run 2: order B, A  → verdict
STEP 4: Verdicts agree → accept. They differ → tie/manual review

Steps 2–3 are separate requests or chats.

Task: An Amazon seller is choosing between two AI-retouched versions of a sneaker photo for a product listing. The instruction was: "Replace the background with white, don't touch the sneaker itself." One version was made by an AI model, the other by an outside contractor. The seller wants an AI judge to help pick the better one. He knows the judge is easily swayed by hints, so he says nothing about which version is "from the contractor" and doesn't mention what his colleagues think.

Prompt (run 1, order A→B):

You are evaluating the result of a product photo retouch.
Edit instruction: "Replace the background with white, don't touch the sneaker itself."
Original photo: [image 1]
Option A: [image 2]
Option B: [image 3]

Rate each option on three criteria from 1 to 10:
1. Instruction following (background is white, object untouched).
2. Edit quality (edges, artifacts, naturalness).
3. Detail preservation (laces, sole, texture, logo).

Then choose: A, B or Tie. Give your answer in this format:
Scores A: ... / Scores B: ... / Verdict: ... / Reason in one sentence.

Run 2 is the same prompt in a new chat with images 2 and 3 swapped: option A is the former image 3, option B is the former image 2. The verdict is compared accounting for the swap.

Result: The model gives three scores per option, a verdict and a short explanation. If the verdicts from both runs point to the same image, the choice is more reliable. If "A" wins the first run and "B" wins the second (i.e., the first slot wins every time), the judge was reacting to order, not quality. That case needs to be decided by eye.

The weakness. A judge model doesn't measure quality in a vacuum. It reads the entire context: image order, captions, other people's opinions, eye-catching details. It can take any such cue as evidence. As a result, the same edit gets different scores depending on what's next to it. "Agrees with humans" is no guarantee either: a judge can match humans on ordinary data and stop matching as soon as an irrelevant cue shows up in the context.

The strength. If you compare the same thing twice, any discrepancy is immediately visible. A judge is free of bias only when its verdict doesn't change when the order is swapped. You can check this with ordinary requests, no access to the model's internals needed.

How the method uses this. A double run in mirrored order turns hidden bias into a visible discrepancy. Agreement between the two runs is a strong signal. Disagreement shows the judge can't be trusted on that pair. Measuring "noise" (repeating the same request) helps separate random scatter from real bias.

Control knobs: - Number of runs → more orders and repeats = more reliable, but more expensive. - Agreement rule → a strict one (both runs match) filters out more pairs. A lenient one (keep a non-tie if the other order gives a tie) keeps more but lets more errors through. - Answer format → an absolute 1–10 score and a pairwise comparison react to the same cues differently. The paper shows that saturation and watermarks shift scores a lot but barely affect pairwise picks. Try both formats and compare. - Context hygiene → strip everything unrelated to quality (model names, other people's opinions, "what's in the image" descriptions) from the prompt.

This is an adaptation of the paper's findings, not a prompt from the paper. The evaluation criteria (three 1–10 dimensions and an A/B/Tie verdict) come from the study design.

You are an independent evaluator of image editing results.
Judge only what is visible in the images. Ignore anyone's opinions, 
model names and captions; you don't have any.

Edit instruction: "{instruction}"
Original image: [source]
Option A: [image 1]
Option B: [image 2]

Rate each option on three criteria from 1 to 10:
1. Instruction following: {what exactly was supposed to change}
2. Edit quality: {edges, artifacts, naturalness}
3. Detail preservation: {what was not supposed to change}

Then choose: A, B or Tie.
Answer format:
Scores A: IF=_, EQ=_, DP=_
Scores B: IF=_, EQ=_, DP=_
Verdict: A / B / Tie
Reason: one sentence.

How to use it: 1. Run the template in one chat with this order (A = image 1, B = image 2). 2. Run it in a new chat with the reverse order (A = image 2, B = image 1). 3. Compare the verdicts, accounting for the swap. Accept the result only if they match. 4. To check for noise, repeat run 1 once more with no changes. If the verdict "drifts" on its own, the judge is unreliable on that pair.

What to fill in: {instruction} is the exact wording of the edit; the three criteria are tailored to your task (what should change, what shouldn't).

🚀 Quick start — paste into a chat:

Here's a template for evaluating image editing results. Adapt it to my task: {describe what you're editing and what you want to achieve}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what exactly the edit instruction was and what in the frame was off-limits, because criteria 1 and 3 depend on that. It will plug this into the template and keep the answer format.

⚠️ Countermeasures are barely tested: The paper shows that biases exist and how strong they are. It treats the two-order double run as a technique (the MT-Bench rule), not as a proven cure. It filters out unstable pairs, but at the cost of coverage: for the "noisiest" model, a lot gets filtered out. Whether a line like "ignore other people's opinions" in the prompt helps was not tested by the authors.

⚠️ Different judges, different weaknesses: The ranking of models changes depending on what you measure: robustness to cues, agreement with humans or stability under swapping. There's no single "best" judge model, and you can't assess reliability with one metric.

⚠️ The effect depends on format: A cue that moves a numeric score may barely affect a pairwise pick, and vice versa. A conclusion about one format can't automatically be carried over to the other.

⚠️ Some cues aren't purely "neutral": The embedded zoom-in inset showed signs during the quality check that it changes the edit itself. The authors treat it as a borderline case.

⚠️ Narrow scope: The study is about evaluating image edits. For text judges and other tasks the same holds "in spirit" (position bias has been known for a long time), but it wasn't measured here.

⚠️ Note: The paper text I was given cuts off at the section on verifying quality preservation. The appendices and final conclusions aren't visible.

The idea was simple: to prove the judge was "swayed," you first have to make sure the cue didn't change the quality of the edit itself. Otherwise you can't tell whether the judge got it wrong or the image really got worse. So the authors took 1,196 real edits from five public benchmarks and added 13 cues at four points in the evaluation process: how it's displayed, the prompt text, the image pixels and the content of the edited region.

Each cue was checked for "quality preservation" in three ways. First: image similarity (SSIM). Second: three separate verifier models, calibrated on an "empty" intervention (SHAM) and on deliberately botched edits. Third: humans, 30 examples per cue. Then five judges (gpt-5.5, gemini-3.5-flash, kimi-k2.5, qwen3.5-plus and VIEScore on GPT-4o) rated edits with and without the cue. Results were compared not against zero but against each judge's own "noise": a repeated request and SHAM.

A few things were surprising. A sticker covering up to 4% of the frame moved scores more than any whole-image manipulation. A fake "majority opinion" inflated all three ratings for every judge. Of 34 significant changes in agreement with humans, 33 were drops, most often with cues tied to the edited region. Swapping A/B flipped up to 60.9% of verdicts (qwen3.5) versus 10.5% for a simple repeat. The practical takeaway: repeating a request won't reveal position bias; you have to actually change the order.

🔧 Technique: "blind judge" → remove sources of bias.

Clean the context before you run it: remove model names, "contractor's version," "last round's winner" and any "what most people think." The paper shows these exact cues (authority, fake majority) moved the scores.

Evaluate the options as anonymous. You don't know where the options came from.
Other evaluators' opinions are not shared with you.

The principle (clean context) comes from the paper. The phrase itself was not tested in the study.

🔧 Technique: measure your own noise → know your threshold.

Run the same pair 3–5 times with no changes. If the score drifts by 1–2 points on its own, don't draw conclusions from a 1-point difference between two options.

Extrapolation: the same double run for text comparisons. The paper is about images, but position bias is known in text judges too. For choosing the better of two texts (two headlines, two sales pitches):

Run 1: "Option A: {text 1}. Option B: {text 2}. Pick the better one by {criterion}."
Run 2: "Option A: {text 2}. Option B: {text 1}. Pick the better one by {criterion}."
If the same text wins both times, accept it. If "the first one" wins both times, that's position, not quality.

This is an extrapolation; it wasn't tested in this paper.

• Paper title: Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation (EditJudgeBias).
• Authors: Yuan Huang (Northeastern University; visiting student at MBZUAI), Zirui Song, Xiuying Chen (Mohamed bin Zayed University of Artificial Intelligence).
• Related work cited in the paper: VIEScore (rubric-based evaluation), CALM (12 biases of LLM judges), MT-Bench (the two-order rule), FRAME, MM-JudgeBias.
• Status: preprint.