MRRG (Multi-Role Rubric Generation): evaluating text from five perspectives at once

STEP 1: Generate the rubric (one prompt)
  → LLM sequentially takes on 5 roles
  → Each role generates 3–7 binary criteria with weights (1/2/3)
  → Duplicates are removed

STEP 2: Evaluate the text (one prompt)
  → LLM checks the text against each criterion (yes/no × weight)
  → Result: weighted score + which criteria failed

Two separate requests. First, build the rubric. Second, run the evaluation.

Task: You wrote a product landing page for a new SaaS service that automates HR paperwork. You want an honest evaluation — not "looks good," but the actual weak spots.

Prompt (Step 1 — generate the rubric):

I have a landing page for an HR SaaS product, here's the text:

---
[PASTE LANDING PAGE TEXT]
---

Now evaluate this text one role at a time, from five roles.
For each role, generate 3–7 specific, binary criteria
(meaning each one can be answered with "yes" or "no").
Assign each criterion a weight: 3 (critical), 2 (important), 1 (polish).

Role 1 — USER: focus on value, clarity of purpose, what the visitor gets
Role 2 — HR EXPERT: focus on accuracy, correctness of claims, trustworthiness of data
Role 3 — EDUCATOR: focus on clarity of explanations, accessibility of language, safety of promises
Role 4 — AI RESEARCHER: focus on accuracy of AI-related claims, honesty about capabilities
Role 5 — LINGUIST: focus on text quality, structure, readability, tone

Format for each role:
**[ROLE NAME]**
- [criterion] [W{weight}]
- [criterion] [W{weight}]
...

After all roles — remove exact duplicates and output the final merged list.

Prompt (Step 2 — evaluation):

Now take the final list of criteria from the previous step
and check the landing page text against it.

For each criterion:
- ✅ if the text meets it
- ❌ if it doesn't — briefly explain what's wrong

At the end: overall score (sum of weights of passed criteria / sum of all weights × 100%)
and the top 3 items to fix first.

Result: Step 1 returns a structured rubric — 15–25 specific criteria broken down by role. Step 2 gives you a checklist with checkmarks/X's, a numeric score, and a prioritized fix list. Instead of a vague "the text is decent but could be better," you get a concrete list: "no evidence for the time-savings claim [W3]," "HR jargon left unexplained [W2]."

LLMs are very good at following role assignments — when a role is defined explicitly and narrowly, the model zeroes in on it. The problem with a plain "evaluate this text" request is that the model picks its own angle — and it's almost always the angle of a "generic educated reader." An educator, a subject-matter expert, and a user each notice different things.

When you ask a single role to cover everything, you get perspective compression: the model effectively averages all viewpoints into one. A narrow role generates sharper, more specific criteria because it isn't trying to "be objective" in some broad sense.

Levers you can pull: - Roles — swap the set to fit the task. For an investor pitch, add "INVESTOR" and drop "LINGUIST" - Criteria range — narrow "3–7" down to "3–5" for a quicker pass - Weights — replace the three-point scale with a binary one (matters / doesn't matter) to simplify - Steps — merge into a single prompt if the text is short and you don't need the extra granularity

I have a {type of material}, here it is:

---
{your text / response / material}
---

Evaluate it one role at a time, from five roles.
For each role, generate 3–7 binary criteria (yes/no answers).
Assign each a weight: 3 (critical), 2 (important), 1 (polish).

Role 1 — USER: {focus of role 1}
Role 2 — EXPERT: {focus of role 2}
Role 3 — EDUCATOR: {focus of role 3}
Role 4 — CRITIC: {focus of role 4}
Role 5 — {custom role}: {focus of role 5}

Format for each role:
**[ROLE]**
- [criterion] [W{weight}]

After that — remove exact duplicates and output the final list.

Then check the {type of material} against each criterion:
✅ meets it / ❌ doesn't meet it + brief explanation

At the end: score in % and the top 3 priorities to fix.

Placeholders: - {type of material} — "landing page," "business email," "project plan," "resume," "sales script" - {role focus} — make it specific to the task: "alignment with market realities," "readability for beginners," "legal risk" - Roles 1–5 — swap them out to fit the evaluation context

🚀 Quick Start — paste into chat:

Here's an MRRG template for multi-role text evaluation.
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask: what text to evaluate, which roles fit the task, which aspects matter — because without that, it can't set each role's focus correctly.

⚠️ Wasted tokens on simple tasks: If the task has a clear-cut answer — "is this formula correct" or "is this grammatically correct" — five roles is overkill. The method pays off on multidimensional material: texts, ideas, plans, strategies.

⚠️ Roles can overlap: For similar tasks, EXPERT and USER may generate overlapping criteria. That's fine — duplicates get removed. But if you see this happening a lot, make your roles more contrasting.

⚠️ Quality depends on how sharply the role is defined: If you just write "Role 2 — EXPERT" without specifying the domain, the criteria will be vague. The sharper the role's focus, the sharper the criteria.

⚠️ Doesn't replace real expertise: An LLM playing "financial risk expert" won't replace an actual financial analyst. This is a structured framework that helps you not miss an angle — not a substitute for depth of expertise.

The researchers asked a simple question: what does an LLM lose when it evaluates text from one perspective instead of several? They compared their multi-role rubric generator against three "single-voice" methods — a plain zero-example baseline, one with an example answer, and the "Chasing the Tail" method (which refines a rubric by comparing different good answers).

They tested it on five models — ranging from a small 3B model to GPT-120B — across three preference benchmarks (RewardBench-2, JudgeBench, PPE). Interestingly, any single role in isolation underperformed even the simple baseline methods — but combining all five roles produced the best results overall. This rules out the simple explanation of "more criteria = better": the researchers specifically tested this by repeating single-voice generation five times, and the result was noticeably worse than the real multi-role version.

A second experiment trained a model using MRRG-based rewards instead of single-voice rewards. Even a small 3B model trained on multi-role signals generalized better to tasks from other domains. The key insight: broader coverage of evaluation criteria → a more robust signal → better generalization.

🔧 Adaptation 1: Roles tailored to a specific profession

The default set (user, expert, educator, AI researcher, linguist) is the paper's baseline. For evaluating a business idea, it's more useful to run:

CUSTOMER → INVESTOR → COMPETITOR → COO → MARKETER

Same principle — five non-overlapping viewpoints. Swap the cast to fit the task.

🔧 Adaptation 2: Self-review before finalizing

Use MRRG not to evaluate a finished result, but before you finalize it. The setup:

Here's a draft of my {material}.
Play these roles one at a time: [5 roles suited to the task].
For each role — find 2–3 weak spots I need to fix.
Then combine everything into a prioritized list of edits.

This isn't a review — it's a pre-flight check before you hit send. Especially useful for emails to partners, pitches, presentations.

🔧 Adaptation 3: Evaluating someone else's content

Competitor teardowns, analyzing other people's landing pages, vetting articles before you share them. MRRG isn't limited to your own material — feed it any text for a structured breakdown.

Paper title: Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling

Authors: Dazhi Fu, Jiuding Yang, Yiwen Guo, Jicong Fan

Organizations: School of Data Science, Chinese University of Hong Kong (Shenzhen); LIGHTSPEED (Tencent)

Code: github.com/fudazhiaka/Many-Voices-One-Reward-Multi-Role-Rubric-Generation-for-LLM-Judging-and-Reward-Modeling

Related work: Chasing the Tail (Zhang et al., 2025), Rubrics as Rewards (Gunjal et al., 2025), MT-Bench / Chatbot Arena (Zheng et al., 2023)