TL;DR
When you ask an LLM to check or grade someone's answer ("is this right," "does it meet the criteria"), the model may invent an extra rule that wasn't in the task, simply because it saw something unfamiliar in the data. It recalls "how it's supposed to look according to the template" and penalizes deviation from that template, not an actual violation of the condition.
A team building automated checking of SQL queries found that their AI judge agreed with human grades on hard cases almost at random, at coin-flip level. It wrongly rejected a correct answer in 77% of cases. The cause turned out to be a single one: the judge saw rare but legitimate values in the data (e.g., codes "E" or "z" instead of the usual "A, B, C, D") and decided they were errors, even though the question never asked to exclude them.
The fix isn't perfect, but it's partial: remove the extra raw-data "preview" from the judge's prompt (this noticeably reduces the effect), and, if you use several AIs as checkers, don't mix a weak judge with a strong one. Instead, take several already-reliable judges and trust the result only when all of them agree, sending disputed cases for manual review.
Method Blueprint
STEP 1: Collect disputed cases where the human and the AI judge disagree → a sample of "hard" examples
STEP 2: Compare human and AI grades on this sample → learn the real % of agreement, not a guess
STEP 3: Break down the reasons for the judge's errors → find the recurring pattern (e.g., "unfamiliar value → invented rule")
STEP 4: Clean up the judge's prompt: remove extra raw data that triggers the pattern → a narrower focus on the real criteria
STEP 5 (if you have several judges): run the check through 2-3 independent models/chats → trust only unanimous verdicts, send disagreements for manual review
Steps 1-2 are a one-time diagnosis (you can do it by hand on 20-30 examples). Steps 4-5 are what you apply in every working prompt.
Example
Task: You keep a marketing report in Google Sheets and ask ChatGPT to check whether an analyst filtered the customer table correctly: "keep only those currently paying for a subscription." Besides the usual "Active" and "Canceled," the "status" column occasionally has legitimate but non-standard values: "Paused," "Pending refund."
Prompt:
You are a reviewer. Assess whether the table below is filtered correctly, using only this criterion:
"Rows with status Active are kept".
Don't add any other requirements. If the data contains rare or unfamiliar
status values ("Paused", "Pending refund", etc.), that is NOT an error by default.
It's an error only if a row with a status other than "Active" remains in the filtered data.
Verdict: COMPLIANT / NOT COMPLIANT.
If NOT COMPLIANT, point to the specific row and explain why exactly it violates the criterion above,
without inventing additional rules.
Data:
{paste the table}
Result: ChatGPT will give a verdict strictly against the stated criterion, without nitpicking rare but valid statuses that the filter didn't touch. If you run the same prompt in Claude and compare the answers, you get a second independent verdict for cross-checking.
Why it works
An LLM judge tends to fill in the template: seeing the data, it recalls what such a set of values "usually" looks like (e.g., grades A-D) and treats any deviation as an error, even if the question didn't ask about it. This isn't a bug of one particular model but a general property: the model is good at predicting the "typical" and bad at telling "atypical but correct" from "atypical, therefore wrong."
An LLM's strength is precisely following explicit, narrow instructions, as long as extra details don't throw it off. By removing the raw data "preview" from the prompt and narrowing the criteria to the literally stated rule, you give the model no reason to compare against a "canonical" template: it just checks what's written.
Control levers: - How much context the judge sees → less raw data = fewer false positives, but also less chance of catching a real error. You'll need to find the balance for your task. - Number of independent judges and the agreement rule → the more judges and the stricter the requirement (all agree), the higher the precision, but more cases go to manual review. - Explicitness of the criteria → the more concretely the rule is stated ("this exact column, this exact value"), the less the model "makes things up."
Prompt template
You are a reviewer (judge). Assess {item_under_review} strictly against the criteria below,
without adding any other requirements.
The task {item_under_review} is supposed to accomplish: {task_statement}
Assessment criteria (these only):
1. {criterion_1}
2. {criterion_2}
3. {criterion_3}
Important: if the data/text contains rare, unfamiliar or non-standard
elements, that is not an error by default. It's an error only if it directly
violates one of the criteria above.
Verdict: COMPLIANT / PARTIALLY COMPLIANT / NOT COMPLIANT.
Name the specific criterion that is violated — without invented rules.
{item_under_review}:
{content}
Fill in: {item_under_review} — what you're checking (a report, text, code, a candidate's answer), {task_statement} — the original question/goal, {criterion_1-3} — the real, explicit conditions of the check, {content} — the material itself.
🚀 Quick start — paste into the chat:
Here is a prompt template for an AI judge that doesn't invent extra rules.
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask about the specific review criteria and what exactly you're checking, because without explicit, narrow criteria the model will start "making up" rules from the template again.
Limitations
⚠️ The fix is incomplete: even the best prompt variant (removing the data preview) reduces false positives but doesn't fully close the gap with a truly calibrated judge.
⚠️ An ensemble hurts when one member is weak: if one of several judges is systematically wrong, adding it to the team makes the overall result worse, not better. First check that each judge is accurate enough on its own.
⚠️ A one-time check doesn't guarantee stability: an AI judge's accuracy changes when the model, version or data type changes, so you need periodic re-checks on fresh human-labeled examples.
⚠️ Removing context can cost you real errors: if you narrow what the judge sees too much, it may miss a genuine problem that required exactly that "extra" piece of data.
How it was studied
The team built a production pipeline that generates its own questions and SQL queries against a telecom company's customer database and ends with a final grade from an AI judge; this pipeline feeds an internal tool used daily by ~30 analysts. When the authors (two PhD researchers) manually compared the judge's verdicts with their own labels on hard cases, agreement was at the level of random guessing, and moderate on a random sample. Breaking down the causes showed that almost all false positives came from one pattern: the judge saw unfamiliar but legitimate values and penalized them.
They then tested four prompt variants, swapped the judge model (for the open Qwen and for top-tier Claude Opus), and tried different ways of combining several judges. It turned out that mixing a weak and a strong judge worsens the result, while three independently reliable judges with the rule "all agree, we trust it; otherwise, to a human" give the best balance of precision and manual-review volume. Testing on public benchmarks (BIRD, Spider) showed that the same error pattern surfaces there too, and it even helped uncover real bugs in "gold-standard" queries written by experts.