Stable "personalities" of model judges

Strictness ranking (strictest to most lenient):

1. Claude-Opus — the strictest (-0.43 from the mean), uniformly picky across all criteria
2. Claude-Sonnet — strict (-0.34), cites a lot but with average citation precision
3. GPT-5.2 — moderately strict (-0.26), especially strict on facts (-0.64)
4. Grok-3 — neutral (±0.00)
5. DeepSeek-R1 — lenient (+0.16), weak fact-checking
6. Mistral-Large — lenient (+0.19), DOESN'T SEE hallucinations
7. Llama-405B — lenient (+0.20), 20% of citations come from content that doesn't exist
8. GPT-4.1 — lenient (+0.21), cites little but precisely (44% precise citations)
9. Gemini-3-Pro — the most lenient (+0.26), best at catching hallucinations

Styles of handling evidence

GPT-4.1 — "The careful critic": - Few citations (7.6 per evaluation) - High precision (44% of citations support the conclusion) - Lenient scores

Claude-Sonnet — "The strict and wordy one": - Many citations (11.8 per evaluation) - Average precision (31%) - Strict scores

Llama-405B — "The sloppy softie": - Few citations (6.1 per evaluation) - 20% of citations are made up (the lowest validity rate, 80%) - Lenient scores

The critical finding on hallucinations

On text with planted fabricated facts:

Mistral-Large and Llama-405B rated the fake HIGHER than the truth. Gemini gave 60% of the fake texts a score of ≤3, while Mistral/Llama/Grok gave 0% failing scores.

An LLM generates text based on patterns in its training data. When a model evaluates (rather than creates), it applies the patterns it learned about "what's good." But those patterns differ from model to model.

Claude was trained with an emphasis on safety and accuracy → the model is strict, especially about risks. Gemini was optimized for a broad audience → the model is lenient and tolerant of variation. GPT-4.1 is a stable older version → conservative in its scores but precise in the details.

Developers don't program "strictness" explicitly — it emerges from the data and the training process: RLHF (reinforcement from human feedback), constitutional AI (behavioral rules), datasets of examples of quality content. Each model absorbs its own implicit "aesthetic of quality."

The stability of the patterns comes down to determinism: given the same input and low temperature, a model produces similar scores. Even at temperature 0.7, variation within a model is smaller than the differences between models. So the disagreements aren't noise — they're persistent differences in approach.

Levers for understanding: - Strictness/leniency — a by-product of training on strict/lenient examples (which scores the model saw during RLHF) - Emphasis on criteria — which aspects were stressed in the training data (facts vs. style vs. structure) - Handling evidence — how well the model was trained to verify claims (including by citing sources) - Hallucination detection — whether there was dedicated training on spotting fakes (Gemini has it, Mistral/Llama don't)

You can't change these properties with an ordinary prompt — they're encoded in the model's weights. But you can CHOOSE the right judge for the task.

1. Picking a judge for the task

Checking facts and hallucinations

Task: Check a client's text for factual errors and exaggerations

Prompt for Gemini-3-Pro:

Check this text for factual errors and exaggerations. 
Rate the reliability of each claim on a 1-5 scale.
For every questionable phrase, quote it and explain the problem.

[paste text]

Why Gemini: It's the best at detecting hallucinations (a -1.46 point drop on fake content). A high share of citations, sensitive to facts.

A strict pre-publication review

Task: Get a strict assessment before publishing a piece

Prompt for Claude-Opus:

Evaluate this piece as a strict editor would before publication.
Criteria (rate each 1-5):
- Factual accuracy
- Completeness of coverage
- Structure and logic
- Readability

Point out everything that needs work. Be picky.

[paste piece]

Why Claude-Opus: The strictest judge (-0.43), uniformly picky across all criteria. If it gives a 4+ → the piece is genuinely solid.

Quick feedback on a draft

Task: A quick read on a draft — get a sense of the overall direction

Prompt for GPT-4.1:

Rate this draft on the key criteria (1-5):
- Clarity of the main idea
- Logical flow
- Persuasiveness

Give 2-3 top tips for improvement.

[paste draft]

Why GPT-4.1: Lenient but careful, cites little (won't bury you in details), high precision of remarks (44%). Good for first-pass feedback without overload.

2. The "different judges" test for critical decisions

Task: Evaluate an important piece — an investor deck, a critical report for leadership, a public statement on behalf of the company

Prompt (run in 2-3 different models):

Evaluate this [piece] against these criteria:
1. Factual reliability (1-5)
2. Strength of the arguments (1-5)
3. Clarity for the target audience (1-5)
4. Risks and weak spots (1-5)

For each criterion:
- Score
- 1-2 quotes from the text to back it up
- A recommendation for improvement

[paste piece]

How to read the results:

• Claude gave 3, Gemini 4.5 → different "theories of quality"; look at the comments, not the numbers
• All three gave 4+ → the piece is objectively solid
• Mistral rated it highest → check the facts separately (Mistral has a blind spot for hallucinations)
• Claude gave 5, the others 3 → the piece may be unconventional; dig into why Claude in particular approved it

❌ Don't average the scores. "3 from Claude + 5 from Gemini = 4" is a synthetic number that reflects the view of none of the judges.

3. Diagnostics: learn a new model's "personality"

Prompt:

Rate these three versions of a text on factual accuracy (1-5):

Version A (control):
[clean text with verified facts — for example, a short summary of a well-known event]

Version B (with a hallucination):
[the same text + 2-3 invented facts — for example, a nonexistent date or person]

Version C (incomplete):
[the same text, but with 40% of the key points left out]

For each version, give a score and 2-3 quotes to back it up.

What to look for: - A–B difference > 1 point → the model catches fakes (like Gemini) - A–B difference ≈ 0 → a blind spot (like Mistral/Llama) - Lots of quotes + a low score → strict and wordy (like Claude-Sonnet) - Few quotes, but precise ones → careful (like GPT-4.1) - 20% of quotes not from the text → sloppy (like Llama-405B)

Evaluate this [piece: text/deck/article] against the following criteria (1-5):

1. {criterion_1} — {description of what we're evaluating}
2. {criterion_2} — {description}
3. {criterion_3} — {description}
4. {criterion_4} — {description}

For each criterion:
- Score (1-5)
- 1-2 specific quotes from the text to back it up
- A brief recommendation for improvement (if the score is < 4)

Overall verdict: is the piece ready to publish, or does it need more work?

[paste piece]

What to fill in: - {piece} — text, deck, article, email, post - {criterion_N} — factual accuracy, persuasiveness, clarity, structure, tone, SEO, fit with the brief - {description} — what exactly this criterion covers

Important: - Choose your judge model deliberately: Claude for strictness, Gemini for fake detection, GPT-4.1 for quick, careful feedback - Don't use Mistral/Llama if the criterion is "factual accuracy" — they're blind to hallucinations - For critical pieces, run it in 2-3 models and compare the comments; don't average the numbers

⚠️ A model ≠ a universal measuring instrument: Different models don't measure "quality in general"; they apply their own hidden criteria. A 4 from Claude and a 4 from Gemini are different 4s.

⚠️ A prompt won't override the model's nature: Telling Gemini "be strict" won't make it as picky as Claude. The baseline calibration is encoded in the model's weights.

⚠️ Averaging does harm: The mean score from 3 judges produces a synthetic verdict that matches the actual position of none of them. Look at the comments and the patterns of disagreement, not the arithmetic mean.

⚠️ Mistral/Llama and hallucinations: These models are blind to fabricated facts (they rated the fake +0.01/+0.27 above the truth). Don't use them to check reliability.

⚠️ Domain specificity: The study covers SEO content and Wikipedia-based briefs. The patterns replicate, but absolute values may shift in other domains (code, creative writing, scientific texts). Relative strictness holds: Claude is stricter than Gemini everywhere.

⚠️ Llama cites things that don't exist: 20% of Llama-405B's citations come from content that isn't in the source material. That's not "imprecision" — it's fabricating evidence.

The researchers ran an experiment: 9 current models (Claude-Opus, Claude-Sonnet, GPT-5.2, GPT-4.1, Gemini-3-Pro, Grok-3, DeepSeek-R1, Llama-405B, Mistral-Large) evaluated the same content against the same rubric.

Material: 30 YouTube videos (various topics: comedy, AI/ML, travel, sports, tech reviews). For each video, 4 versions of SEO packs (descriptions, tags, titles) were generated by different LLM generators (GPT-5.2, GPT-4.1, Gemini-3-Pro, Mistral/Claude). That's 120 unique pairs (video + pack). Each pair was evaluated by each model 3 times independently → 3,240 evaluations.

Rubric: 5 criteria (1-5 points): intent, completeness, factual accuracy, readability, SEO mechanics. The models returned structured JSON with scores + quotes from the source as justification. Strict format compliance was required (invalid JSON → excluded from analysis). Models with compliance <98% were dropped (Cohere, Kimi K2).

What they measured:

1. Agreement between judges (Krippendorff's α) — how much the models agree with each other
   → Result: α = 0.04 (nearly zero, versus a norm of >0.67)

2. Within-judge stability (ICC) — how much a model agrees with itself across 3 runs
   → Result: a range from -0.04 to 0.87; the top models are very stable

3. Attribution — can you tell from the scores which model produced them
   → A Random Forest classifier, grouped cross-validation by video

The results were surprising:

Agreement between models is almost zero (α = 0.04). On two criteria (readability, SEO) agreement is negative (α = -0.06 and α = -0.05) — that's worse than chance, meaning a systematic divergence: when one model gives a high score, another tends to give a low one.

But within themselves, many models are very stable: Gemini ICC=0.87, GPT-5.2 ICC=0.85, Claude-Opus ICC=0.81. This means the disagreements aren't noise but persistent differences in how they approach evaluation.

The classifier guessed the model from its scores with an accuracy of: - 77% (from just the 5 rubric numbers) - 90% (adding meta-features: how many quotes, how many are valid, whether the quotes connect to the conclusions) - 99.6% for telling GPT-4.1 and GPT-5.2 apart (same provider, different versions) — almost perfect

This means every model has a unique signature in its evaluations.

Cross-domain check:

They repeated the study on different material — 15 Wikipedia articles → briefing packs with 8 sections (TL;DR, key ideas, timeline, FAQ, glossary, etc.). They deliberately planted controlled defects: - Hallucination-poisoned: 3-5 planted fake facts - Coverage-poisoned: 40-50% of key subtopics left out - Structure-poisoned: broken structure (missing sections, wrong format)

The patterns replicated: Claude strict, Gemini lenient, Mistral/Llama blind to fakes. Attribution accuracy was 90.3% — same as with YouTube, despite a third of the data (1,066 vs. 3,240 evaluations). Telling GPT-4.1 and GPT-5.2 apart on Wikipedia content: 100% (perfect accuracy).

The hallucination test (the most striking finding):

They compared how models rate clean text vs. text with fakes. Gemini lowered its factual-accuracy score by -1.46 points, GPT-5.2 by -1.12, Claude-Sonnet by -0.92. But Mistral-Large moved +0.01 (didn't notice the problem), and Llama-405B moved +0.27 (rated the fake HIGHER than the truth).

Even more dramatic: Gemini gave 60% of the fake texts a score of ≤3 (a fail). Mistral, Llama, Grok: 0% failing scores for fakes. They're blind.

Artifact checks:

• Permutation test: shuffled the judge labels → accuracy dropped to 8% (chance) ✓
• Tokens-only probe: removed every feature except text length → accuracy ≈ chance ✓
• Leave-one-video-out: validation with no video overlap between train/test → accuracy fell to 60% but stayed significant (vs. 11% chance) ✓
• Temperature test: varied the temperature (0.0, 0.3, 0.7) → patterns stable; within-model differences smaller than between-model ones ✓
• Marginal stripping: z-score and quantile normalization that strip per-judge calibration → fingerprints persist ✓

Conclusion: the signal is real — not an artifact of text length, overfitting to specific videos, or chance.

💡 Adaptation for prompt benchmarking:

If you're developing a prompt and want to test its quality across models, don't average the scores — look at the pattern of disagreement.

I have a prompt for generating product descriptions for an online store.
Generate 3 versions of a description for [product: e.g., "Sony WH-1000XM5 wireless headphones"].

Then rate each version on these criteria:
1. Appeal to the shopper (1-5)
2. Accuracy of the specs (1-5)
3. SEO optimization (1-5)

For each score, quote a passage to back it up.

Run it in Claude and Gemini. If both give 4+ → the prompt is objectively solid. If Claude gives 3 and Gemini 5 → study Claude's comments (it's stricter, so its criticism carries more weight for product content).

💡 Adaptation for RLHF / training on AI feedback:

If you plan to train a model on scores from an LLM judge (reinforcement learning from AI feedback), the choice of judge is critical — it determines which patterns the model will absorb.

Task: Train a model to write email newsletters for your product.

❌ Bad:
Use Gemini as the judge → the model learns lenient criteria and will let weak phrasing slide

✅ Good:
Use Claude-Opus as the judge → the model learns strict criteria for accuracy and structure

✅✅ Even better:
Use an ensemble: Claude for facts, GPT-4.1 for persuasiveness, Gemini for catching blatant errors → the model learns multidimensional criteria

Don't think of an LLM judge as a neutral grader. It's a teacher passing on its own aesthetic.

🔧 Technique: Prompt-based calibration → a partial shift

The study shows that baseline strictness/leniency is encoded in the model's weights. But you can shift the calibration by 10-20% with a prompt:

Rate the text on a 1-5 scale.

Calibration: I've noticed you tend to give [high/low] scores.
For this task, apply a [stricter/more lenient] standard.

Reference points:
- 5 = exceptional, top 5% of material
- 4 = solid, above average
- 3 = acceptable, meets baseline expectations
- 2 = weak, needs work
- 1 = unacceptable

[criteria and text]

This won't override the model's nature, but it can shift absolute values by 10-20%. Crucially, the pattern of relative priorities stays (Claude is still stricter on facts, Gemini softer overall). Use it for fine-tuning, not for replacing the judge.

💡 Extrapolation: Multi-judge consensus for contested decisions

For critical decisions (launching an ad campaign, a public statement, a contract), use a multi-judge protocol with explicit resolution of disagreements:

[Run in 3 models: Claude-Opus, Gemini-3-Pro, GPT-4.1]

Evaluate this [piece] against these criteria:
1. Reputational risk (1-5)
2. Factual accuracy (1-5)
3. Fit with the brand's tone of voice (1-5)

For each criterion: a score + justification with quotes.

[piece]

Then a meta-prompt to resolve the disagreements:

I have 3 evaluations of the same piece from different experts:

Expert A (strict): [paste Claude's scores and comments]
Expert B (fake detector): [paste Gemini's scores]  
Expert C (careful): [paste GPT-4.1's scores]

Analyze where they agree, where they diverge, and why.
Give a final recommendation: publish / revise / reject.

If revise, list specific edits for each type of criticism.

This gives you structured resolution of disagreements instead of synthetic averaging.

Evaluative Fingerprints: Stable and Systematic Differences in LLM Evaluator Behavior
Author: Wajid Nasser (Viore), January 2026

The study builds on: - MT-Bench and Chatbot Arena (Zheng et al., 2023) — the foundations of LLM-as-judge, the first large-scale benchmarks with LLM graders - G-Eval (Liu et al., 2023) — rubric-based evaluation with GPT-4 - CALM framework (Ye et al., 2024) — a taxonomy of bias in LLM judges, principles for diagnosis - Haldar & Hockenmaier (2025) — self-inconsistency in LLM-as-a-judge, within-model variability - Behavioral Fingerprinting (Pei et al., 2025) — characterizing models through diagnostic prompts