STEP 1: Create the evaluation plan
├─ Option A (heuristic): use a ready-made template for the task type
├─ Option B (self-reflection): the model analyzes the task and defines criteria
└─ Option C (combo): combine heuristics and self-reflection

→ A plan that separates primary/secondary criteria

STEP 2: Run the evaluation
Evaluate the texts strictly according to the plan from Step 1 → final verdict

Both steps can be done in one prompt or split into two requests.

Task: Pick the best startup pitch for an accelerator. You have two options: a short, punchy one (150 words) vs. a detailed, structured one (450 words). You need an evaluation without length bias — reasoning models tend to overrate longer texts.

Prompt:

You are an accelerator expert. Evaluate two startup pitches and pick the best one.

STEP 1 — EVALUATION PLAN:
Analyze the task and lay out a plan:
- Which criteria are PRIMARY (decisive for the choice)?
- Which are SECONDARY (nice to have, but not critical)?
- How do you avoid the trap: overrating length, structure over substance?

Write the plan out explicitly.

STEP 2 — EXECUTION:
Evaluate the pitches strictly according to the plan from Step 1:
- Primary criteria first
- Then secondary ones
- Final choice

PITCH A (short):
{text of pitch A — 150 words}

PITCH B (detailed):
{text of pitch B — 450 words}

Result: The model will produce an explicit plan with criteria (e.g., "Primary: the problem being solved, business model, competitive advantage. Secondary: level of detail, structure, visual elements"). Then it will go through each item in the plan for both pitches. The final choice will be justified through the plan, not through "pitch B is more detailed and structured" (the classic length bias).

The weakness of reasoning models: During a long chain of thought, the model systematically checks the answer from different angles. It sees length, structure, specificity — and those surface metrics overshadow the substance. The model reasons: "Answer B has 8 points, 3 examples, a clear structure" → inflates the score even if it's weaker in substance. The bias is amplified precisely by reasoning.

The strength of reasoning models: They're excellent at following explicit instructions and executing multi-step procedures. Give them a clear algorithm ("first this, then that, ignore this") and the model will carry it out meticulously. In the study, reasoning models showed 8-12% better adherence to evaluation instructions than their base versions.

How PlanJudge harnesses that strength: An explicit plan separates primary and secondary criteria ("what matters is accuracy; details are secondary"). The reasoning model locks in that priority at the start and checks back against it at every step of the evaluation. When the model gets to "answer B is longer," it remembers the plan: "length is a secondary criterion, back to accuracy." The bias stops working.

Control levers:

• Plan generation: heuristic (fast, fine for standard tasks) vs. self-reflection (flexible, for complex cases) vs. combo (maximum effect, +20-30% tokens)
• Criteria granularity: more criteria → more accurate evaluation, but slower; fewer → faster, but you might miss something important
• Explicit anti-biases: you can write "DON'T factor in length" or "ignore the number of examples" into the plan — the model will lock it in as a rule
• Splitting into requests: plan in one chat, evaluation in another → a fresh context for execution, reducing bias carryover from planning

Option A: Combined (maximum effect)

You are an expert in {evaluation domain}. Evaluate {what you're evaluating} and pick the best option.

STEP 1 — CREATE THE EVALUATION PLAN:

Account for the specifics of the task:
{task heuristic — see below}

Additionally, analyze:
- Which criteria are PRIMARY (decisive)?
- Which are SECONDARY (desirable)?
- Which evaluation traps should be avoided (length, concreteness, structure bias)?

Write out the final evaluation plan explicitly.

STEP 2 — RUN THE EVALUATION:

Evaluate the options strictly according to the plan from Step 1:
1. Check the primary criteria for each option
2. Check the secondary criteria
3. Final decision, justified through the plan

OPTION 1:
{text of the first option}

OPTION 2:
{text of the second option}

Heuristics by task type:

• Creative writing (articles, posts, copy): "Primary: value of the idea, benefit to the reader, clarity. Secondary: style, length, formatting."
• Analysis (breakdowns, research, reports): "Primary: depth of analysis, soundness of conclusions, factual accuracy. Secondary: data completeness, visualization."
• Code: "Primary: correctness of the solution, readability, efficiency. Secondary: comments, style."
• Ideas/solutions (business, product, strategy): "Primary: solves the problem, feasibility, uniqueness. Secondary: level of detail, presentation."

Placeholders: - {evaluation domain} — your field: marketing, development, business, content - {what you're evaluating} — specifically: startup pitches, post variants, solutions to a problem - {task heuristic} — pick from the list above or write your own - {text of the option} — paste the texts being evaluated

Option B: Fast (heuristic only)

If you need speed and the task is standard, use only the heuristic plan without self-reflection:

You are an expert in {domain}. Evaluate {what} according to the following plan:

EVALUATION PLAN:
Primary criteria: {criteria from the heuristic}
Secondary criteria: {criteria from the heuristic}
Ignore: {what not to factor in}

Evaluate the options strictly according to the plan:

OPTION 1:
{text 1}

OPTION 2:
{text 2}

🚀 Quick start — paste into the chat:

Here's a PlanJudge template for reducing bias in evaluation. Adapt it to my task: {describe what needs evaluating and the criteria}.
Ask questions to fill in the fields.

[paste the template above]

The LLM will ask: what you're evaluating, what type of task it is (creative/analysis/code/ideas), which criteria matter most, and which biases you need to guard against. These questions help the model tune the plan to your specifics — it'll take the PlanJudge structure and build in your priorities. You'll end up with a ready-made prompt with an evaluation plan tailored to your task.

⚠️ Length: PlanJudge adds 20-30% more tokens to the request. The combined option (heuristic + self-reflection) is the longest but the most effective. For simple tasks, use the fast option (heuristic only).

⚠️ The effect is stronger on reasoning models: PlanJudge works on ordinary LLMs too (reducing bias by 7-11%), but the maximum effect (a 16-32% bias reduction) is on reasoning models like DeepSeek-R1, QwQ-32B, Qwen3-Thinking. They follow multi-step instructions better.

⚠️ Not a cure for every bias: PlanJudge works great against surface biases (length, concreteness, structure). But if both options are objectively close in quality, the model may still choose unpredictably.

⚠️ You need clear criteria: If the task is subjective and the criteria are fuzzy ("rate the creativity"), a plan won't help. PlanJudge shines where you can explicitly separate primary from secondary.

A team from the Harbin Institute of Technology and the Institute of Science Tokyo took 4 pairs of models: a reasoning version vs. a base version from the same family (DeepSeek-V3 vs. R1, Qwen2.5-32B vs. QwQ-32B, Qwen3-30B vs. Thinking-2507, Qwen3-Next-80B vs. Thinking). They tested 4 key aspects of a judge's work.

Evaluation accuracy: Tested on RewardBench (1700+ examples from chat, code, math, safety) and JudgeBench (various domains). Reasoning models beat the base versions by 1-2% on average, but the jump came on hard tasks — code and math. There the gap reached +8-12%. Why? A reasoning model can solve such problems itself through a long chain of thought, so it catches subtleties in other people's solutions. Surprisingly, on easy tasks (knowledge, chat) the difference is minimal — long reasoning gives no advantage when judging simple answers.

Instruction following: The researchers built Helpsteer2-trivial — a dataset with a trap. They take pairs of answers where answer A is better overall, but answer B is better on a specific dimension (e.g., helpfulness). The model gets two prompts: "evaluate overall" and "evaluate strictly on helpfulness." If the model switches from A to B between the prompts, it's following instructions correctly. Reasoning models showed a Reversal Rate of 91-97% vs. 83-88% for base models. Conclusion: reasoning models apply specific evaluation criteria more meticulously, even when that contradicts the overall impression.

Robustness to attacks: Tested on RobustJudge — 8 types of prompt injection attacks (from simple "ignore previous instructions" to sophisticated "fake reasoning" and "empty suffix"). Reasoning models turned out to be 50-100% more robust (the iSDR metric — lower is better). Why? During reasoning, the model repeatedly checks alignment with the original task. The injection stands out as an anomaly: "Wait, I'm supposed to be evaluating answers, not following a new instruction." Base LLMs generate the answer faster → fewer checks → easier to hijack.

Bias: Here's the surprise. On BiasBench (length and concreteness bias), reasoning models flopped — 15-17% worse than base models. But on LLMBar (where the answer clearly fails to follow the instruction), they were 3-10% more robust. The insight: reasoning models are great at catching logical violations (off-topic answers, ignored conditions) but get fixated on metrics (length, structure, detail). The chain of thought turns into a checklist: "the answer has 8 points, 3 examples, it's structured" → an automatic high score.

PlanJudge was tested on all the benchmarks — the combined method delivered +16-32% on BiasBench while preserving accuracy on RewardBench. The simplicity of the fix was striking: no fine-tuning, no external tools — they just added a planning step to the prompt.

The researchers didn't provide a single template — they described three approaches to generating the plan. Below are the key elements from their prompts:

Heuristic plan (Heuristic-based):

For creative writing tasks:
Primary: Idea value, reader benefit, clarity
Secondary: Style, length, formatting

For analytical tasks:
Primary: Analysis depth, conclusion validity, factual accuracy
Secondary: Data completeness, visualization

For code evaluation:
Primary: Solution correctness, readability, efficiency
Secondary: Comments, style conventions

Self-reflective plan (Self-synthesized):

Step 1: Analyze the current evaluation task
- What is being evaluated?
- What are the PRIMARY criteria (decisive)?
- What are the SECONDARY criteria (desirable)?
- What biases should be avoided (length, concreteness, structure)?

Step 2: Formulate an explicit evaluation plan

Running the evaluation (Execution phase):

Execute the evaluation according to the plan:
1. Evaluate primary criteria for each response
2. Evaluate secondary criteria for each response
3. Make final decision based on the plan

Context: The researchers tested all three variants separately, then created a combined one (heuristic + self-reflection), which gave the best result. For real work they recommend the combo for important tasks where errors are costly, and heuristic only for standard tasks where you need speed.

💡 Adaptation for comparing your own drafts

The classic case: you've written 2-3 versions of a text (a post, an article, an email) and can't choose. A plain "which is better?" prompt gives a vague answer or a bias toward the longest one. PlanJudge forces the model to explicitly prioritize criteria based on your goal.

You are an editor. Help me pick the best version of a LinkedIn post.

CONTEXT:
Goal of the post: {attract clients / show expertise / go viral}
Audience: {description}

STEP 1 — PLAN:
Based on the goal and audience:
- Which qualities of the post are PRIMARY?
- Which are SECONDARY?
- Which bias should you guard against (length, number of paragraphs, emoji)?

STEP 2 — EVALUATION:
Evaluate the 3 versions strictly according to the plan:

VERSION 1:
{text}

VERSION 2:
{text}

VERSION 3:
{text}

Final choice + explanation through the plan.

Effect: The model won't say "version 2 is more detailed and structured" (the classic bias). It'll say: "The goal is to show expertise. Primary: insight + justification. Secondary: structure. Version 1 delivers a strong insight in 150 words; version 2 stretches to 400 without adding anything new. Choice: version 1."

💡 Adaptation for hiring / evaluating candidates

Reasoning models tend to overrate detailed résumés and lengthy take-home assignments. A candidate can write a lot of low-value text and outscore a concise expert. PlanJudge locks in what's primary: solving the problem and quality of thinking — not volume of text.

You are a hiring expert for the {job title} position. Evaluate how two candidates completed a take-home assignment.

ASSIGNMENT:
{task description}

STEP 1 — PLAN:
For the {job title} position, which qualities of the solution are:
- PRIMARY (a dealbreaker if missing)?
- SECONDARY (a plus, but not critical)?
- Which metrics should NOT influence the choice (volume of text, number of examples, visual formatting)?

STEP 2 — EVALUATION:
Evaluate the solutions strictly according to the plan:

CANDIDATE A:
{solution text}

CANDIDATE B:
{solution text}

🔧 Technique: An anti-bias checklist in the plan → protection against a specific bias

If you know what specifically the reasoning model inflates (length, structure, number of examples), write it into the plan as an explicit prohibition. The model will lock it in as a rule and ignore it during evaluation.

STEP 1 — PLAN:
...
ANTI-BIAS CHECKLIST (do NOT factor in when evaluating):
- Length of the answer in words / characters
- Number of points / paragraphs / structural elements
- Presence of examples (judge the QUALITY of examples, not the count)
- Visual formatting (emoji, headings, lists)

These metrics are secondary or don't matter at all. Base the decision on {primary criteria}.

Effect: While reasoning, the model will run into this checklist and correct itself: "Answer B has 8 examples... wait, the plan says to ignore the count — looking at quality."

🔧 Technique: Splitting into two requests → a fresh context for execution

You can split PlanJudge: the plan in the first chat, the execution in the second. This reduces the risk that the model carries bias over from analyzing the task into the evaluation itself.

Chat 1 (planning):

Task: evaluate {what}.
Goal: {why}.

Create an evaluation plan:
- Primary criteria
- Secondary criteria
- Anti-bias checklist

Plan only, no evaluation.

Chat 2 (execution):

Here's the evaluation plan:
{copy the plan from the first chat}

Evaluate the options strictly according to this plan:

OPTION 1:
{text}

OPTION 2:
{text}

Why: In the first chat, while building the plan, the model may accidentally form a preference for a certain type of answer. A fresh chat wipes that context — the model sees only the finished plan and the texts being evaluated.

Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases — Hui Huang, Xuanxin Wu, Muyun Yang, Yuki Arase (Harbin Institute of Technology, Institute of Science Tokyo, The University of Osaka)

Research code: https://github.com/HuihuiChyan/LRM-Judge