MERG (Metacognitive Enhanced Rubric Generation): Forcing Knowledge Activation Before Evaluation — How to Get Honest Criticism from an LLM Instead of Compliments

MERG — 4 steps in one prompt, in order:

STEP 1 (Knowledge Activation) 
  → The LLM lays out what it knows about the domain:
    quality standards, typical mistakes, expert criteria

STEP 2 (Metacognitive Reflection)
  → The LLM names its likely biases:
    "I may overrate formatting / a confident tone / length"
    + strategies to avoid them

STEP 3 (Dynamic Rubric Generation)
  → Based on steps 1-2, the LLM builds 
    a rubric specific to this task
    (not "style / structure" but "does it build an atmosphere of psychological tension?")

STEP 4 (Calibrated Evaluation)
  → Scores each criterion with quotes from the text
  → Final check: did any of the biases from step 2 leak in?

All four steps in one prompt, one request.

Task: Jake Morgan wrote a landing page for his crypto trading course. He asked ChatGPT to rate it and got 8.5/10, with praise for its "clear structure" and "compelling CTA." But the copy promises "guaranteed returns of 40% a month" — a direct violation of advertising law and of the rules of most payment processors.

A standard request won't catch it. MERG will.

Prompt:

Evaluate my landing page for a crypto trading course. 
Go through four steps in order, without skipping any.

STEP 1 — DOMAIN KNOWLEDGE:
Before you look at the text, lay out what you know 
about the online trading education industry:
- What regulatory and legal restrictions exist 
  for advertising financial products?
- Which promises in this niche break platform rules 
  (Facebook/Instagram Ads, Google Ads, TikTok) or FTC advertising rules?
- Which trust standards actually matter in this niche?
- What separates a legitimate course from a shady scheme?

STEP 2 — POTENTIAL BIASES:
Name how you might be biased when evaluating this text:
- Which surface features (formatting, confident tone, 
  a clear CTA, length, "magic numbers") could create 
  the illusion of good copy even if it's weak in substance?
- Which red flags do you tend to underrate?

STEP 3 — RUBRIC:
Based on steps 1-2, build a rubric specific to evaluating 
this exact type of landing page. Not generic "style" and 
"structure" criteria, but the criteria that matter for a legitimate 
landing page for a financial course.

STEP 4 — EVALUATION:
Score the text against the rubric from step 3.
For each criterion, give specific quotes from the text.
At the end: did any biases from step 2 leak in?

Here's the landing page copy:
[paste your text]

Result:

In step 1 the model lays out what it knows about the niche: FTC rules on advertising financial services, Facebook and Google Ads policies on "guaranteed income," and how legitimate education differs from shady schemes.

In step 2 it admits: "I tend to rate a confident tone, specific return figures and a clear call to action highly — even when they break the rules."

In step 3 it builds a rubric with criteria like "Legal safety of claims" and "Compliance with platform ad policies" — instead of a bland "Copy quality."

In step 4 it flags "guaranteed returns of 40%" as a direct violation and gives concrete wording fixes, rather than just praising the "persuasive style."

Why standard evaluation misleads. LLMs learned from huge swaths of internet text, where "good writing" is often associated with clear structure, a confident tone and professional formatting. That's a pattern of surface features the model has learned as a "quality" heuristic. When you ask it to "rate my text," the model fires up this fast pattern — and a text with slick layout and confident phrasing gets a high score even if the content is wrong or dangerous.

Why agreement between several models doesn't save you. The study found something paradoxical: the higher the agreement between three different judges, the more likely they're all relying on the same surface heuristics. Three different AIs from three different companies — and all three missed an illegal business model while unanimously praising the formatting. That's the illusion of consensus.

How MERG breaks the pattern. Step 1 forces the model into slow mode: instead of "recognize a familiar pattern" it's "first tell me what you know about this domain as an expert." Step 2 makes biases visible before they kick in. When the model itself says "I tend to overrate a confident tone," it's less likely to fall for it. It's like asking a judge to declare a conflict of interest before ruling.

Control knobs:

• Detail in step 1 → the more precise the knowledge domain, the sharper the evaluation. Instead of "finance," write "FTC rules for advertising credit products in the US, 2024"
• Number of criteria in step 3 → 3-4 criteria for a quick check, 7-8 for a deep dive
• Final bias check → without it step 2 loses its point, don't remove it
• Specific quotes in step 4 → drop the quote requirement and the model will slide back into generic phrases

Evaluate {what to evaluate} — {short description for context}.
Go through four steps in order.

STEP 1 — DOMAIN KNOWLEDGE:
Before you look at the material, lay out:
- What do you know about quality standards in {domain}?
- What are the typical mistakes and red flags for this type of {material}?
- What criteria does a real expert in {domain} use 
  to tell good from bad?

STEP 2 — POTENTIAL BIASES:
Name how you might be biased when evaluating this material:
- Which surface features could create an illusion of quality?
- What do you tend to overrate or underrate?
- How will you compensate for it?

STEP 3 — RUBRIC:
Based on steps 1-2, create a rubric specific 
to evaluating this exact type of material.
Not generic categories, but concrete criteria that matter in {domain}.

STEP 4 — EVALUATION:
Score the material against the rubric from step 3.
For each criterion: a specific quote or example from the text.
At the end: did any biases from step 2 leak into the evaluation?

Material to evaluate:
{text/description}

Placeholders: - {what to evaluate} — "a pitch," "a landing page," "an article," "a business plan," "a resume" - {short description for context} — "for a B2B SaaS in HR tech," "for a product manager opening" - {domain} — "online education," "legal services," "CPG marketing" - {text/description} — your material

🚀 Quick start — paste into a chat:

Here's a template for the MERG method (evaluation with knowledge activation). 
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask: what exactly we're evaluating, in which domain and for what purpose — because steps 1 and 3 need a concrete domain; without it the rubric will be generic and pointless. The model will take the pattern from the template and adapt it to your task.

⚠️ Subjective domains: When evaluating literature, creative writing or design, MERG lowers agreement — and that's correct (it exposes real disagreements). But if you need one clear answer rather than "a range of opinions," expect a less clear-cut result.

⚠️ Limits of the model's knowledge: MERG only works in domains the LLM knows deeply enough. For very narrow specialties (radiation safety standards, rare legal precedents) the model may activate incomplete or outdated knowledge in step 1 — which creates false confidence.

⚠️ Model level vs. specific text: The study showed that LLMs reliably distinguish quality categories (weak / average / strong). But at the level of a specific text, accuracy is lower. MERG improves things but doesn't fix them completely. Critical decisions still require a human expert.

⚠️ More tokens: Four detailed steps make for a long answer. In free tiers with limits it may get cut off. Use a paid plan or split it into two requests.

A team from Tencent asked: if three strong LLMs independently evaluate the same text and almost always agree, does that mean they understand quality? Or are they just reacting to the same surface signals?

To find out, they ran 32 different language models (from weak base models to strong "thinking" ones like DeepSeek-R1) through evaluation — 100 diverse tasks (from creative writing to academic texts), three judges (Claude 4.5 Opus, Gemini 2.5 Pro, GPT-5.1), at 11 different temperature settings. In total — 105,600 evaluation instances.

The key trick: they took the MERG method and compared scores with and without forced knowledge activation. If the judges' original agreement reflected real understanding, MERG shouldn't have broken it. If the agreement was superficial, MERG should have split it apart.

The result was inconvenient: knowledge broke the consensus strongly and predictably (–21 to –34% agreement, with large effect sizes of d=0.97–1.42). And it broke it differently: in the hard sciences and education, agreement went up — because there are shared standards that knowledge anchors the judges to. In literature it went down, because real aesthetics has no single right answer. That's the clever part: noise would break agreement equally everywhere. The fact that the pattern differed across domains is direct evidence that this is an illusion, not random interference.

The most surprising part: the better the model, the lower the agreement between judges when evaluating it. Weak base models get scored unanimously (the surface signs of poor quality are obvious to everyone). Strong "thinking" models get maximum disagreement. Which means that exactly where evaluation matters most (sorting the best of the best), LLM judges are least reliable.

The researchers describe the logic of MERG's four stages in the context of evaluating a gothic story writing task. Here's how the rubric changes:

WritingBench (standard):
1. Style and Creativity
2. Coherence and Structure
3. Language Quality
4. Content Relevance
5. Overall Impression

MERG for Gothic Horror:
1. Atmospheric Dread Construction 
   — Does the text create genuine psychological tension 
     through environmental and psychological dread, 
     not just genre trappings?
2. The Uncanny Familiar 
   — Are ordinary elements defamiliarized in ways 
     that create unease rather than comfort?
3. Narrative Agency of Terror 
   — Does the horror emerge organically from the story's 
     logic rather than being imposed?
4. Gothic Tradition Engagement 
   — Does the text demonstrate knowledge of and 
     creative dialogue with gothic conventions?
5. Linguistic Atmospheric Control 
   — Does the prose rhythm, diction, and syntax 
     reinforce rather than undermine the atmosphere?

Context: This comparison shows Stage 3 of the method in action — how the same task gets different criteria under knowledge-grounded evaluation. "Style" turns into a concrete question about how exactly narrative horror is being built.

💡 Adaptation: reviewing a resume before an important application

A standard "check my resume" prompt gets you comments on formatting and spelling. MERG will surface the mismatch between your experience and the requirements of the specific job.

Evaluate my resume for a {role title} position at {company type}.
Go through four steps.

STEP 1 — KNOWLEDGE:
What do you know about hiring for this role?
- What do companies really look for in {role title} candidates at {company type}?
- Which red flags in a resume get it rejected immediately?
- How do recruiters read resumes (where do they start, where do they lose interest)?

STEP 2 — BIASES:
How might you be biased?
- What looks good on the surface in a resume but doesn't matter for this role?
- What's easy to miss?

STEP 3 — RUBRIC:
Build evaluation criteria for this specific type of resume.

STEP 4 — EVALUATION:
Score it with quotes. Check for biases.

Resume:
{resume text}

Job description:
{job posting text}

🔧 Technique: multi-perspective MERG evaluation

If you want several points of view, run MERG several times with different roles in step 1, specifying concrete experts:

[Same prompt, but at the start:]
"You are a seasoned venture investor focused on B2C startups in the US. 
Everything you know is this expert's knowledge."

Run it three times: investor → potential customer → competitor. You'll get three fundamentally different rubrics and evaluations instead of an averaged-out opinion.

🔧 Technique: "step 1 only" for a quick briefing

Sometimes the first step alone is enough — as context activation before any question about an unfamiliar domain:

Before answering, lay out what you know about {domain}:
quality standards, typical traps, red flags.
[then ask your main question]

This works for any question where domain expertise matters, not just for evaluation.

Paper: Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge (Preprint, 2025)

Authors: Mingyang Song, Mao Zheng, Chenning Xu — Large Language Model Department, Tencent, China

Related work: - WritingBench (Wu et al., 2025) — the benchmark used as the dataset (WritingBench) - G-Eval (Liu et al., 2023) — a GPT-based evaluation method - MT-Bench (Zheng et al., 2023) — the original LLM-as-a-Judge benchmark - Kahneman (2011) "Thinking, Fast and Slow" — the conceptual basis for System 1/System 2