Apply to my task
arXiv:2606.06754 ★ 90 Jun 4, 2026

MADRAG: Three Roles to Beat the LLM's "Average" Answer

Key result

LLM judges tend to regress toward the middle of the scale and avoid extreme scores. Our guess at why: pros and cons cancel each other out during generation. MADRAG gets you calibrated scores that genuinely distinguish "good" from "excellent", with no fine-tuning, just prompts plus a bank of already-scored examples. Three roles force the viewpoints apart: the Advocate talks only about strengths, the Skeptic only about weaknesses, and the Judge gets both arguments plus reference examples at different levels to compare against. References matter most: removing them cut the share of scores within 1 point of the human grade by about 22–24 points, removing debate by about 3–4 points (GPT-5, student essays).

01

TL;DR

LLMs grade with a pull toward the middle: the paper notes that LLM judges regress toward the middle of the scale and avoid extreme scores even when they are deserved. As a result, "good" and "excellent" work can end up with similar scores.

One reason is that the model has no external anchor. It produces a score from general impressions rather than by comparing against concrete examples of strong and weak work. Our guess at a second reason: a single agent hunts for pros and cons at the same time, they cancel each other out, and the verdict slides to "fine."

MADRAG fixes this in two moves: it splits the search for strengths and weaknesses across separate roles (Advocate → Skeptic → Judge), and it retrieves already-scored example essays at different score levels for the Judge (one per score point, from a bank of human-scored essays). The Advocate talks only about strengths. The Skeptic, only about weaknesses. The Judge sees both views plus the reference examples and delivers a calibrated score.

Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.

02

Method Blueprint

STEP 1: ADVOCATE: strengths only → a written argument (no score)
STEP 2: SKEPTIC: weaknesses only, taking the Advocate's argument into account → 
        a written rebuttal (no score)  
STEP 3: JUDGE: reads both arguments + reference examples at different levels → 
        final score with justification

In the paper, each role is a separate model call, and only the Judge sees
the examples. Fitting all three into one prompt is our shortcut (not tested).
03

Example

Our illustration: the paper tested only student essays, and the reference pitches below are ours.

The task: Jake Turner has written an investor pitch for his B2B SaaS product. He wants to know how convincing the pitch really is, not just "yeah, it's fine." The usual "rate my pitch" request tends to return a safe middle score like 7/10.

Prompt:

You are an expert pitch evaluation system. Play three roles in sequence.

Here is the pitch:
"[paste the pitch text]"

Evaluation criteria: persuasiveness for an investor (clarity of the problem, 
market size, uniqueness of the solution, team, traction).
Scale: 1–10.

---

ROLE 1: ADVOCATE
Find only the strengths of the pitch. 
Don't mention weaknesses. Don't give a score. Only arguments in favor.

[Advocate:]

---

ROLE 2: SKEPTIC
Read the Advocate's arguments and the pitch itself.
Find only the weaknesses: what isn't convincing, what raises doubts, 
what's missing.
Don't mention strengths. Don't give a score. Only arguments against.

[Skeptic:]

---

ROLE 3: JUDGE
Read both arguments. Use the reference examples below for calibration.

Reference 1 (score 2/10): "We're building an app for everyone. The market is huge. 
We have no competitors."
Reference 2 (score 5/10): "We solve reporting automation 
for small businesses. The market is $5B. There are 3 competitors; we're faster. 
No customers yet; we launch next quarter."
Reference 3 (score 9/10): "200 paying customers, $120K MRR, 
15% month-over-month growth. We solve a CFO pain point at companies with 50–500 employees. 
Competitors solve an adjacent problem, not this one. Team: ex-CFO from Stripe + ex-CTO from Intuit."

Compare the pitch to the references. Give a score on the 1–10 scale 
with a concrete justification: why not higher and why not lower.

[Judge:]

Result: The model will produce three blocks in sequence. The Advocate lists concrete strengths. The Skeptic lists pinpoint weaknesses, building on the Advocate's arguments. The Judge gives a score positioned "relative to the references", for example: "6/10: strong team and a clear problem, but no traction and a fuzzy ICP. Closer to reference 5, slightly higher for team quality." Not "fine", but a specific position on the scale.

04

Why It Works

The problem (our explanation; the paper only documents the pull toward the middle): an LLM grading head-on sees pros and cons at the same time, and they may get averaged in the middle of generation, pulling the answer toward the center of the scale.

What models do well: play roles with hard constraints. Tell it "strengths only, no weaknesses" and the model faithfully stays inside the frame. It can argue within set boundaries instead of just generalizing.

How the method exploits this: splitting into Advocate and Skeptic forces the viewpoints apart. Each role generates a deliberately one-sided but honest argument. The Judge receives views that are already separated instead of a mush, which makes it easier to deliver a weighted judgment rather than an average one. References at different points of the scale provide an anchor: the model compares instead of guessing an absolute score. In the paper's ablations, the references (retrieval) did most of the calibration work; debate added a smaller refinement.

Control knobs (our suggestions; the paper used one example per score point and scored each trait separately, but did not compare these options):

  • Number of references → at least 3 (low / medium / high). You can add 5 for finer calibration
  • Role strictness → tighten the ban: "Advocate: if you mention a weakness, rewrite" holds the frame even harder
  • Criteria → the Judge scores one criterion more accurately than five at once. Split into separate requests per criterion
  • Judge output format → add "explain why not 1 point higher and not 1 point lower" to pull out concrete boundary reasoning
05

Prompt Template

You are a system for evaluating {what we are evaluating}.

Here is the material to evaluate:
"{text}"

Criteria: {evaluation criteria}.
Scale: {minimum}–{maximum}.

---

ROLE 1: ADVOCATE
Strengths only. No scores. No weaknesses.
[Advocate:]

---

ROLE 2: SKEPTIC
Weaknesses only, taking the Advocate's arguments into account. 
No scores. No strengths.
[Skeptic:]

---

ROLE 3: JUDGE
Read both arguments. Use the references for calibration:

Reference for {low_score}: "{example of weak work}"
Reference for {mid_score}: "{example of average work}"
Reference for {high_score}: "{example of strong work}"

Compare against the references. Give the final score with a justification: 
why not 1 point higher and why not 1 point lower.
[Judge:]

What to fill in: - {what we are evaluating}: a pitch, a sales proposal, an article, a script - {evaluation criteria}: specific ones, such as "persuasiveness for the client", "clarity of structure", "concreteness of arguments" - {example of weak/average/strong work}: real examples. Use actual texts you've seen, or ask the LLM to generate references before running the method (the paper used real human-scored essays; generated references were not tested)


🚀 Quick start: paste this into the chat:

Here is the MADRAG template for evaluating texts. 
Adapt it to my task: {your task}.
Ask me questions to fill in all the fields.

[paste the template above]

The LLM will ask what to evaluate, by which criteria, and for reference examples, because without anchors on the scale the Judge can't calibrate the score. It will take the template's structure and assemble a working prompt for your task.

06

Limitations

⚠️ Weak on countable criteria: grammar, spelling, number of errors: debate doesn't help here and can even add noise. On the Conventions trait, the version without debate sometimes scored better, and all LLM variants trailed a feature-based trained model (QWK 0.19–0.28 vs 0.62 on essay set 7).

⚠️ Weaker without references: if you don't give the Judge examples at different levels of the scale, you lose most of the calibration. The study found the reference examples are the main driver: removing them cut the share of scores within 1 point of the human score by about 22–24 points (e.g. 0.795 → 0.575 on essay set 7), while removing debate cut it by about 3–4 points.

⚠️ Subjective taste criteria: "I like the style" without clear criteria: the method won't help. You need observable features you can put into words.

⚠️ Cost and data: three roles plus references make for a big context, and the paper needed several model calls per trait plus a bank of already-scored essays. It was tested on only two narrative essay sets, so transfer to other genres is unproven. For very short tasks (rate a single paragraph) a direct prompt is simpler.

07

How They Tested It

A team from the University of California, Irvine took the standard ASAP dataset of student essays graded by human examiners (essay sets 7 and 8, the only ones with per-trait scores) and asked a tough question: can a system with no training get close to the accuracy of trained neural networks?

They tested MADRAG on GPT-3.5-turbo, GPT-4o-mini, GPT-4o, GPT-5-mini and GPT-5. They compared it against two kinds of systems: ones trained on the dataset (ProTACT, FeatEng-RF) and other training-free prompting approaches (ZS-LLM, CSR-J). The metric was Quadratic Weighted Kappa: roughly, how closely the system's score matches a human examiner's.

The most interesting part surfaced in the ablation tests (checking "what happens if we remove one component"). It turned out that debate and references do different jobs. References fix the pull toward average scores: the system stops being afraid to give a 2 or a 9. Debate improves reasoning about content criteria (ideas, argument structure) but does not help with mechanical ones (grammar, punctuation) and sometimes made them worse. Overall MADRAG beat other prompt-based methods and came close to trained systems, though a feature-based trained model still led on several traits.

An unexpected result: MADRAG with GPT-5 on the "Ideas" criterion exceeded the agreement between human examiners (QWK 0.75 versus 0.69 on essay set 7; 0.67 versus 0.53 on set 8). In other words, on this one trait its scores matched the reference grades more closely than two human graders matched each other. These are single runs, without variance estimates.

08

Adaptations and Extrapolations

🔧 Adaptation: generate references before scoring

Don't have ready-made examples for your criteria? Ask the LLM to create them:

Before scoring, create three references for the criterion "{criterion}":
— Reference 2/10: a text that clearly fails on this criterion
— Reference 5/10: a text that passes but has no real strengths  
— Reference 9/10: a text that wins convincingly on this criterion

Then score my text by comparing it to these references.
My text: {text}

The model first builds the anchors, then scores relative to them. This is our extrapolation: the paper used real scored essays, and self-made references may calibrate less well.


🔧 Adaptation: Advocate and Skeptic as a team before a decision

The role-splitting principle isn't just for scoring. Use it before important decisions:

I am making a decision: {decision}.

ADVOCATE: find only the arguments in favor. Don't mention risks.
[Advocate:]

SKEPTIC: find only the arguments against. Don't mention upsides.
[Skeptic:]

ARBITER: weigh both lists. What would a veteran angel investor 
who has seen 500 startups decide, and why?
[Arbiter:]

Works for: choosing between job offers, launching a new product, changing your positioning. The Advocate defends the idea, the Skeptic tries to kill it, and the Arbiter makes the balanced call.

09

Resources

MADRAG: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring

Authors: Ali Keramati, Shiyuan Zhou, Sharad Mehrotra, Mark Warschauer Organization: University of California, Irvine Contact: [email protected]

ASAP dataset: https://www.kaggle.com/c/asap-aes/data

Related work cited in the paper: ChatEval (Chan et al., 2023), MAGIC (Jordán et al., 2025), CAFES (Su et al., 2025)

Research digest

Key result

LLM judges tend to regress toward the middle of the scale and avoid extreme scores. Our guess at why: pros and cons cancel each other out during generation. MADRAG gets you calibrated scores that genuinely distinguish "good" from "excellent", with no fine-tuning, just prompts plus a bank of already-scored examples. Three roles force the viewpoints apart: the Advocate talks only about strengths, the Skeptic only about weaknesses, and the Judge gets both arguments plus reference examples at different levels to compare against. References matter most: removing them cut the share of scores within 1 point of the human grade by about 22–24 points, removing debate by about 3–4 points (GPT-5, student essays).

Principle

Don't ask for an "objective score" in a single question; split the viewpoints into roles with hard bans. A standard "rate this" request makes the model see pros and cons at once, and (our hypothesis) they can cancel out before a number even appears. The funny part: a hard ban in a role ("strengths only; if you mention a weakness, rewrite") works like blinders on the model, and it faithfully holds the frame. The Advocate and Skeptic hand the Judge already polarized arguments instead of a mixed-up mush. The Judge sees the contrast and compares against references instead of pulling an absolute score out of thin air.

Why it works

An LLM generates a score token by token. When "strong structure" and "weak arguments" are floating in the context at the same time, the next number token may gravitate to the middle of the scale (our hypothesis; the paper documents the effect, not the mechanism). Splitting into roles means the Judge sees already polarized opinions instead of an averaged context. Add references at different points on the scale (the paper used one per score point; in its ablations they did most of the calibration), and the model shifts from "guess an absolute score" to "compare with known points." A far more solvable task.

When to apply

Evaluating texts, pitches, sales proposals, articles, scripts → especially when it matters to distinguish levels, not just "good enough or not." Ideal for situations where a plain "rate this" request gives a dull 7/10 to everything. NOT a fit for: countable criteria (spelling, number of errors; an Advocate and Skeptic don't count errors any better than one attentive agent), very short texts (for a paragraph, a direct prompt is simpler), subjective taste judgments without clear criteria.

Mini recipe

1. Define the task: what you're evaluating and by which criteria. Be specific: not "quality" but "persuasiveness for an investor" or "clarity of structure." 2. Prepare three references: weak / average / strong. Real examples are best. No real ones? You can ask the LLM to generate them first (our workaround; the paper used real scored essays). 3. Run the Advocate: `Strengths of [text] only. No scores. No weaknesses. If you mention a weakness, rewrite.` 4. Run the Skeptic: `Read the Advocate's arguments and the text itself. Weaknesses only: what isn't convincing, what's missing. No scores. No strengths.` 5. Run the Judge: pass both arguments + references at different levels. Ask for a score with a justification of "why not one point higher and not one point lower"; that question pulls out concrete boundary reasoning instead of a bare number.

Examples

[BAD]: `Rate my pitch on a scale from 1 to 10` [GOOD]: `Play three roles in sequence. ROLE 1: ADVOCATE: only the strengths of the pitch below. No scores. No weaknesses. [Advocate:] ROLE 2: SKEPTIC: read the Advocate's arguments and the pitch itself. Weaknesses only: what isn't convincing, what's missing. No scores. No strengths. [Skeptic:] ROLE 3: JUDGE: read both arguments. Reference 2/10: "We're building an app for everyone, the market is huge, there are no competitors". Reference 5/10: "Reporting automation for small businesses, a $5B market, there are competitors, no customers yet". Reference 9/10: "200 paying customers, 15% month-over-month growth, we solve a specific CFO pain point". Compare the pitch to the references. Give a 1–10 score with a justification: why not 1 point higher and why not 1 point lower. [Judge:] Pitch: [paste text]`

In plain words

The problem is that modern language models are terrible conformists. Ask ChatGPT to rate your text on a 1-to-10 scale and it will often land on a safe 7. This is called the pull toward the middle: LLM judges avoid extreme scores even when they are deserved. As a result, a brilliant long read and a lazy throwaway can get similar scores. One plausible reason (our guess, not the paper's): "average" is the most common and safest answer in the model's training data.

It's like hiring a judge who gives every defendant exactly five years, just to avoid offending anyone or taking responsibility. Technically the job is done and a sentence is handed down, but there's zero justice in it. A regular LLM sees pros and cons at the same time, they may blend together, and out comes gray mush. It rarely shows radical criticism or genuine enthusiasm in its scores.

To cure this dullness, researchers came up with MADRAG. The idea is simple: instead of one indecisive grader, you use three roles. One agent looks only for strengths, another only for flaws, and a third, the Judge, weighs both and gets already-scored example essays from a database via RAG to compare against. In the paper, those examples did most of the work against the "everything is a 7" effect; the debate added a smaller boost, mainly on content traits like ideas and organization.

The method was tested only on grading student essays (two narrative essay sets). We expect the principle to carry over to other tasks that need calibrated ranking: from scoring investor pitches to screening resumes or picking the best ad creative, but that is untested. Where a regular prompt says "well, it's fine", debate plus scored examples can better separate gold from junk. We go from a simple question to a full-blown trial inside the model.

Bottom line: if you want honest feedback, don't just ask one model for a number; give it examples at different score levels and let opposing roles argue first. A lone LLM without anchors tends to overrate mediocrity. MADRAG turns AI from a polite neighbor into a tough critic who isn't afraid to hand out ones and tens. Anchors plus a clash of opinions help you get a result that better reflects quality instead of just filling a slot in a report.

AI lab assistant

Apply this study to your task

Describe what you're working on. AI takes the method and the measured results from this paper and adapts them to your case, or writes a ready prompt built on them.

Based on MADRAG: Three Roles to Beat the LLM's "Average" Answer arXiv:2606.06754

Sign in to apply this study New accounts get 50 free credits.