TL;DR
RRD (Recursive Rubric Decomposition) is a method for building detailed evaluation criteria (rubrics) for LLM judges. The idea: broad criteria are recursively broken down into specific sub-criteria until each one is narrow enough to tell answers apart. Then contradictory and duplicate criteria are filtered out, and the remaining ones get weights that account for correlations.
Basic LLM-generated rubrics hurt evaluation accuracy: GPT-4o with simple rubrics drops from 55.6% to 42.9% agreement with humans — 13 points worse than with no criteria at all. The reason: broad criteria like "the text is helpful" or "the answer is clear" fit almost every option and don't help distinguish quality. They add noise instead of signal. On top of that, rubrics often duplicate each other ("clarity" and "lucid presentation" are essentially the same), artificially amplifying some aspects at the expense of others.
RRD works in three stages: (1) An LLM proposes initial rubrics based on the task and 8 sample answers. (2) Recursion: if a rubric fits more than 2 answers, it's too broad and gets split into sub-criteria. Then filtering: criteria with inverted logic (they prefer the weak model over the strong one) and duplicates are removed. The cycle repeats until no new valid criteria emerge. (3) The criteria are assigned weights that account for correlations, so similar criteria don't dominate. Result: ~20 specific, non-overlapping criteria instead of 7 generic ones.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
RRD runs outside the chat (preparing the criteria), then gets applied in the chat:
STAGE 1: Initial criteria
The LLM receives: the task + 8 sample answers (from different models)
→ Generates ~7 initial rubrics
STAGE 2: Recursive decomposition + filtering (loop)
For each rubric:
If it fits >2 answers → split it into sub-criteria
Check the new rubrics:
- Remove if they prefer a weak answer over a strong one
- Remove if they duplicate existing ones
Repeat while valid new rubrics keep appearing
Stop: when 15 proposals in a row are rejected
STAGE 3: Weighting
Assign weights that account for correlations between criteria
→ Final set of ~20 detailed criteria
APPLYING IT IN A CHAT:
Score the answers against these criteria + aggregate the scores
Example
Task: You need to pick the better of two investor pitch decks for a startup that delivers ready-made meals to remote workers.
Prompt (a simplified version of the RRD logic):
I'll give you 2 versions of an investor pitch. Help me pick the better one.
First, create evaluation criteria:
1. Generate 5-7 initial criteria for a good pitch
2. For each criterion, check: if BOTH versions meet it, split it into more specific sub-criteria that tell them apart
3. Remove duplicate criteria
4. Show the final list of criteria
Then score both versions against these criteria and tell me which is better.
[Version A]: ...
[Version B]: ...
Result:
The model will first show the initial criteria (for example: "a clear problem", "hard numbers", "a unique solution"). Then it will split the broad ones into narrow ones: instead of "hard numbers" → "market size in dollars", "unit economics", "revenue forecast". It will remove duplicates ("clarity" / "lucid presentation" → one criterion). It will show a final list of 15-20 specific criteria. Then it will score each version on each criterion and give a reasoned conclusion with a tally.
Why It Works
LLMs are bad at evaluating against generic criteria because they're fuzzy: "persuasiveness" or "helpfulness" can be interpreted dozens of ways. The model picks an arbitrary interpretation that may not match your intent. Plus, generic criteria often fit both options, turning the evaluation into a coin toss.
LLMs are good at working with specific, measurable features: whether there's numerical data, whether a specific risk is mentioned, the order of the arguments. The more specific the criterion ("is there a 3-year revenue forecast"), the more consistently the model checks it.
RRD uses recursive decomposition as a magnifying glass: it starts with broad criteria, then deliberately hunts for differences. The logic is like medical diagnostics: if an "inflammation" test comes back positive for every patient, it's useless — you need tests for the specific type of inflammation. If the criterion "offers a solution to the problem" is met by both pitches, split it into "technological novelty", "barriers to copying", "speed of adoption". Decomposition continues until the criteria start telling the options apart.
Filtering and weighting prevent overfitting to artifacts: criteria with inverted logic are removed (otherwise the model may prefer the worse option), and similar criteria get a lower combined weight (otherwise "good grammar" + "no errors" + "language quality" would triple the weight of one aspect at the expense of the others).
Control levers:
- Split threshold (>2 answers) — lower it to >1 and the criteria get even more specific, but there will be explosively many of them. Raise it to >3 and it's faster but less detailed
- Number of samples (8 answers) — more samples → richer coverage of cases, but more expensive. Fewer → faster, at the risk of missing aspects
- Source of samples — generate them with different models (GPT + Gemini) → more varied styles, deeper criteria. One model → may miss blind spots
- Stop threshold (15 rejections) — higher → dig deeper, but with diminishing returns. Lower → faster, but you may not dig deep enough
Prompt Template
Evaluate {number} candidate answers to a task using recursive criteria decomposition.
TASK: {task_description}
CANDIDATES:
{candidate_answers}
PROCESS:
1. INITIAL CRITERIA
Propose 5-7 criteria for a good answer to this task
2. RECURSIVE DECOMPOSITION
For each criterion:
- Check how many candidates meet it
- If >2 candidates — split the criterion into 2-4 more specific sub-criteria
- Keep splitting until the criteria discriminate between candidates
3. FILTERING
- Remove criteria that defy common sense
- Remove duplicate criteria (keep one of the similar ones)
4. FINAL EVALUATION
Show the final list of criteria
Score each candidate on each criterion (yes/no or points)
Tally up and explain which candidate is better
ANSWER FORMAT:
[Initial criteria]
[Decomposition process with explanations]
[Final criteria]
[Score table]
[Conclusion]
What to fill in:
- {number} — how many options to compare (2-4 is ideal)
- {task_description} — the specific task or question
- {candidate_answers} — a numbered list of candidate answers
🚀 Quick start — paste into a chat:
Here's an RRD template for evaluating options. Adapt it to my task: [your task].
Ask clarifying questions if needed.
[paste the template above]
The LLM will ask which options you're comparing and what matters in the evaluation — because different tasks call for different criteria (technical accuracy vs. persuasiveness vs. creativity). It'll take the recursive decomposition logic from the template and tailor the criteria to your context.
Limitations
⚠️ Computational cost: A single evaluation takes several iterations of generating and checking criteria — 3-5 LLM calls instead of one. Pricey for one-off evaluations; it pays off for systematic ones (when the same criteria are applied to hundreds of options).
⚠️ Requires sample answers: The method works better when there are real examples to analyze rather than an abstract task description. If you don't have any, you'll first need to generate a few candidate answers and then build criteria from them.
⚠️ Subjectivity remains: RRD makes criteria more specific but doesn't solve the fundamental problem — for open-ended tasks (creative work, style, persuasiveness) there's no objective truth. Two people can have different preferences on the same criteria. The method helps formalize YOUR evaluation system; it doesn't find a universally correct one.
⚠️ Overfitting to the samples: If the samples used to build the criteria are too similar (all from one model, one style), the criteria may not work on other kinds of answers. You need variety in the samples.
Why It Works (In More Depth)
The study rests on an elegant theory: the probability of an LLM judge's error drops exponentially as the signal-to-noise ratio of the criteria improves. Signal is how strongly a criterion points to the better answer. Noise is the correlations and contradictions between criteria.
Four principles for minimizing error:
1. Decomposition increases coverage through specialization. A broad criterion like "the answer is helpful" gives a weak signal across many dimensions. Splitting it into "solves the original problem" + "gives concrete steps" + "accounts for constraints" yields 3 independent signals, each stronger in its own zone. The total signal grows.
2. Positivity removes criteria with inverted logic. If a criterion systematically prefers the worse option, its weight is negative and it reduces the final signal. The filter checks against a weak vs. a strong model: if a criterion fires more often on clearly bad answers, it gets tossed.
3. Non-redundancy fights duplication. If "good grammar", "no errors" and "language quality" are three criteria about the same thing, they correlate at ~0.8-0.9. That doesn't triple the signal, but it does triple their weight when summed. The noise in such criteria is also correlated and gets amplified. Solution: keep one or lower the weights.
4. Weighting accounts for correlations mathematically. If we don't know the true importance weights (no human labels), the optimal strategy is whitened uniform weighting: transform the criteria into an uncorrelated space and give them all equal weights there. It's like PCA: strip out the repeated information, keep the orthogonal components. Result: each criterion makes an independent contribution, and correlations don't distort the aggregation.
A medical analogy captures the intuition: a doctor doesn't diagnose holistically ("the patient looks sick"). She orders tests. If a test is positive for many diagnoses, it's broad and needs follow-ups. "Elevated temperature" → could be infection, autoimmune disease, cancer. She orders "inflammation markers" → positive. That narrows it down to inflammatory processes. Next come a "bacterial culture" and "autoantibodies" — these distinguish infection from autoimmune disease. The recursion goes deeper until the tests clearly distinguish diagnoses. Plus the doctor knows: CRP and ESR both measure inflammation, and counting both as independent "votes" is a mistake — it's one signal through different tests.
RRD carries this logic over to text evaluation: start with general quality "symptoms", drill down with specific "test" criteria, stop when the criteria start clearly distinguishing the options, and keep in mind that some criteria are just different measurements of the same aspect.
How They Tested It
The team tested RRD on two human-preference benchmarks: JudgeBench (350 pairs of GPT-4o answers to hard tasks) and PPE (10,200 pairs from Chatbot Arena, 20 models, 121 languages). They compared the LLM judge's accuracy against the human verdict.
Basic LLM rubrics (simply asking the model to create criteria without samples) tanked GPT-4o's accuracy from 55.6% to 42.9% — worse than chance on some subsets. The reason turned out to be that abstract criteria add noise, not signal.
RRD with sample answers, recursion and filtering raised accuracy to 73.3% on JudgeBench (+17.7 points) and 73.7% on PPE. The same held for the open Llama3.1-405B: up from 57.4% to 64.8%.
Interesting: how the number of criteria grew. It started at ~7 and plateaued at ~20 criteria after 3 iterations. But the spread across tasks widened — simple tasks stopped at 12-15 criteria, complex ones reached 30. This shows adaptivity: the method automatically adjusts its depth to the task's complexity.
A non-obvious result: the weighting of criteria matters more than their number. A baseline with 20 criteria but naive averaging lost to RRD with the same 20 criteria but whitened weighting (accounting for correlations). The gap was 3-5 points of accuracy. This confirms that the problem isn't just coverage, but also how the signals are aggregated.
Ablation on the sample source: GPT-4o as the criteria generator works better when the sample answers come from another strong model (Gemini) rather than from itself. Accuracy 73.3% (GPT+Gemini) vs. 71.2% (GPT only). Hypothesis: a variety of answer styles reveals more facets of quality, and the model doesn't get stuck on its own artifacts.
Next, they used the criteria as a reward signal to train Qwen3-4B and Llama3.1-8B on 4K prompts from WildChat. Baseline methods (simple rubrics, iterative ones) gave ~10-20% reward improvement over training. RRD gave 160% for Qwen and 60% for Llama — nearly an order of magnitude stronger. Models trained with RRD improved on the downstream benchmarks BiGGen Bench and HealthBench-Hard, including out-of-domain medical dialogues.
The study's conclusion: a criterion's simplicity is inversely proportional to its value. The more specific and narrow a criterion, the more reliably it works. Recursion is a way to systematically find sufficient granularity. The key is to do it on real samples; otherwise the criteria are detached from practice and do harm.
Adaptations and Extrapolations
🔧 Technique: Two rounds of evaluation → watch the reasoning evolve
In the original, RRD prepares criteria and then runs a one-time evaluation. You can adapt it for debugging reasoning: round one uses generic criteria, round two the decomposed ones. Comparing the verdicts shows which aspects were missed at first.
Evaluate two options in TWO ROUNDS:
ROUND 1: Generic criteria
Create 5 generic criteria and score both options
ROUND 2: Detailed criteria
Take each criterion from Round 1
Split it into 2-3 sub-criteria if both options meet it
Re-score both options on the detailed criteria
Show:
- The Round 1 verdict
- The Round 2 verdict
- What changed and why
This shows where the conclusion came from: if the verdict flipped, the extra detail uncovered an important hidden aspect.
🔧 Technique: Generate criteria BEFORE creating options → steer the thinking
Original: sample answers → criteria → evaluation. You can invert it: criteria → samples → generating your own answer.
TASK: {your_task}
STAGE 1: Create criteria without samples
What criteria separate an excellent answer from a mediocre one?
STAGE 2: Generate 3 samples
- A bad sample (violates most criteria)
- An average sample (meets some)
- An excellent sample (meets all)
STAGE 3: Refine the criteria
Look at the samples. Split broad criteria into specific ones.
STAGE 4: Write the final answer
Use the detailed criteria from Stage 3 as your guide
This is scaffolding for thinking: first we formalize what "good" means, then we create to that spec.
🔧 Technique: Criteria per judge role → specialized evaluation
Basic criteria are universal. You can specialize who is judging:
Evaluate the pitch from the perspective of three judges:
JUDGE 1: Venture investor
Criteria: scalability, unit economics, market size, defensibility
JUDGE 2: Operations expert
Criteria: realistic timelines, team, execution risks
JUDGE 3: A customer from the target audience
Criteria: clear value, desire to buy, price
For each judge:
1. Create criteria specific to their role
2. Split broad criteria into specific ones
3. Score the options
4. Give a verdict on the judge's behalf
Final: An overall verdict that weighs all three perspectives
It's like a multi-agent evaluation, but through the lens of role-specific criteria: an investor doesn't see the same aspects a user does.
Resources
Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks — Tan et al. (2024) JudgeBench, Frick et al. (2024) PPE benchmark, Liu et al. (2025) Dr.GRPO algorithm | Hashemi et al. (2024) rubric-based judging baseline, Gunjal et al. (2025) Rubrics-as-Rewards paradigm | the researchers use GPT-4o, Llama3.1-405B, Gemini 2.5-pro