TL;DR
LLMs grade with a pull toward the middle: the paper notes that LLM judges regress toward the middle of the scale and avoid extreme scores even when they are deserved. As a result, "good" and "excellent" work can end up with similar scores.
One reason is that the model has no external anchor. It produces a score from general impressions rather than by comparing against concrete examples of strong and weak work. Our guess at a second reason: a single agent hunts for pros and cons at the same time, they cancel each other out, and the verdict slides to "fine."
MADRAG fixes this in two moves: it splits the search for strengths and weaknesses across separate roles (Advocate → Skeptic → Judge), and it retrieves already-scored example essays at different score levels for the Judge (one per score point, from a bank of human-scored essays). The Advocate talks only about strengths. The Skeptic, only about weaknesses. The Judge sees both views plus the reference examples and delivers a calibrated score.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: ADVOCATE: strengths only → a written argument (no score)
STEP 2: SKEPTIC: weaknesses only, taking the Advocate's argument into account →
a written rebuttal (no score)
STEP 3: JUDGE: reads both arguments + reference examples at different levels →
final score with justification
In the paper, each role is a separate model call, and only the Judge sees
the examples. Fitting all three into one prompt is our shortcut (not tested).
Example
Our illustration: the paper tested only student essays, and the reference pitches below are ours.
The task: Jake Turner has written an investor pitch for his B2B SaaS product. He wants to know how convincing the pitch really is, not just "yeah, it's fine." The usual "rate my pitch" request tends to return a safe middle score like 7/10.
Prompt:
You are an expert pitch evaluation system. Play three roles in sequence.
Here is the pitch:
"[paste the pitch text]"
Evaluation criteria: persuasiveness for an investor (clarity of the problem,
market size, uniqueness of the solution, team, traction).
Scale: 1–10.
---
ROLE 1: ADVOCATE
Find only the strengths of the pitch.
Don't mention weaknesses. Don't give a score. Only arguments in favor.
[Advocate:]
---
ROLE 2: SKEPTIC
Read the Advocate's arguments and the pitch itself.
Find only the weaknesses: what isn't convincing, what raises doubts,
what's missing.
Don't mention strengths. Don't give a score. Only arguments against.
[Skeptic:]
---
ROLE 3: JUDGE
Read both arguments. Use the reference examples below for calibration.
Reference 1 (score 2/10): "We're building an app for everyone. The market is huge.
We have no competitors."
Reference 2 (score 5/10): "We solve reporting automation
for small businesses. The market is $5B. There are 3 competitors; we're faster.
No customers yet; we launch next quarter."
Reference 3 (score 9/10): "200 paying customers, $120K MRR,
15% month-over-month growth. We solve a CFO pain point at companies with 50–500 employees.
Competitors solve an adjacent problem, not this one. Team: ex-CFO from Stripe + ex-CTO from Intuit."
Compare the pitch to the references. Give a score on the 1–10 scale
with a concrete justification: why not higher and why not lower.
[Judge:]
Result: The model will produce three blocks in sequence. The Advocate lists concrete strengths. The Skeptic lists pinpoint weaknesses, building on the Advocate's arguments. The Judge gives a score positioned "relative to the references", for example: "6/10: strong team and a clear problem, but no traction and a fuzzy ICP. Closer to reference 5, slightly higher for team quality." Not "fine", but a specific position on the scale.
Why It Works
The problem (our explanation; the paper only documents the pull toward the middle): an LLM grading head-on sees pros and cons at the same time, and they may get averaged in the middle of generation, pulling the answer toward the center of the scale.
What models do well: play roles with hard constraints. Tell it "strengths only, no weaknesses" and the model faithfully stays inside the frame. It can argue within set boundaries instead of just generalizing.
How the method exploits this: splitting into Advocate and Skeptic forces the viewpoints apart. Each role generates a deliberately one-sided but honest argument. The Judge receives views that are already separated instead of a mush, which makes it easier to deliver a weighted judgment rather than an average one. References at different points of the scale provide an anchor: the model compares instead of guessing an absolute score. In the paper's ablations, the references (retrieval) did most of the calibration work; debate added a smaller refinement.
Control knobs (our suggestions; the paper used one example per score point and scored each trait separately, but did not compare these options):
- Number of references → at least 3 (low / medium / high). You can add 5 for finer calibration
- Role strictness → tighten the ban: "Advocate: if you mention a weakness, rewrite" holds the frame even harder
- Criteria → the Judge scores one criterion more accurately than five at once. Split into separate requests per criterion
- Judge output format → add "explain why not 1 point higher and not 1 point lower" to pull out concrete boundary reasoning
Prompt Template
You are a system for evaluating {what we are evaluating}.
Here is the material to evaluate:
"{text}"
Criteria: {evaluation criteria}.
Scale: {minimum}–{maximum}.
---
ROLE 1: ADVOCATE
Strengths only. No scores. No weaknesses.
[Advocate:]
---
ROLE 2: SKEPTIC
Weaknesses only, taking the Advocate's arguments into account.
No scores. No strengths.
[Skeptic:]
---
ROLE 3: JUDGE
Read both arguments. Use the references for calibration:
Reference for {low_score}: "{example of weak work}"
Reference for {mid_score}: "{example of average work}"
Reference for {high_score}: "{example of strong work}"
Compare against the references. Give the final score with a justification:
why not 1 point higher and why not 1 point lower.
[Judge:]
What to fill in:
- {what we are evaluating}: a pitch, a sales proposal, an article, a script
- {evaluation criteria}: specific ones, such as "persuasiveness for the client", "clarity of structure", "concreteness of arguments"
- {example of weak/average/strong work}: real examples. Use actual texts you've seen, or ask the LLM to generate references before running the method (the paper used real human-scored essays; generated references were not tested)
🚀 Quick start: paste this into the chat:
Here is the MADRAG template for evaluating texts.
Adapt it to my task: {your task}.
Ask me questions to fill in all the fields.
[paste the template above]
The LLM will ask what to evaluate, by which criteria, and for reference examples, because without anchors on the scale the Judge can't calibrate the score. It will take the template's structure and assemble a working prompt for your task.
Limitations
⚠️ Weak on countable criteria: grammar, spelling, number of errors: debate doesn't help here and can even add noise. On the Conventions trait, the version without debate sometimes scored better, and all LLM variants trailed a feature-based trained model (QWK 0.19–0.28 vs 0.62 on essay set 7).
⚠️ Weaker without references: if you don't give the Judge examples at different levels of the scale, you lose most of the calibration. The study found the reference examples are the main driver: removing them cut the share of scores within 1 point of the human score by about 22–24 points (e.g. 0.795 → 0.575 on essay set 7), while removing debate cut it by about 3–4 points.
⚠️ Subjective taste criteria: "I like the style" without clear criteria: the method won't help. You need observable features you can put into words.
⚠️ Cost and data: three roles plus references make for a big context, and the paper needed several model calls per trait plus a bank of already-scored essays. It was tested on only two narrative essay sets, so transfer to other genres is unproven. For very short tasks (rate a single paragraph) a direct prompt is simpler.
How They Tested It
A team from the University of California, Irvine took the standard ASAP dataset of student essays graded by human examiners (essay sets 7 and 8, the only ones with per-trait scores) and asked a tough question: can a system with no training get close to the accuracy of trained neural networks?
They tested MADRAG on GPT-3.5-turbo, GPT-4o-mini, GPT-4o, GPT-5-mini and GPT-5. They compared it against two kinds of systems: ones trained on the dataset (ProTACT, FeatEng-RF) and other training-free prompting approaches (ZS-LLM, CSR-J). The metric was Quadratic Weighted Kappa: roughly, how closely the system's score matches a human examiner's.
The most interesting part surfaced in the ablation tests (checking "what happens if we remove one component"). It turned out that debate and references do different jobs. References fix the pull toward average scores: the system stops being afraid to give a 2 or a 9. Debate improves reasoning about content criteria (ideas, argument structure) but does not help with mechanical ones (grammar, punctuation) and sometimes made them worse. Overall MADRAG beat other prompt-based methods and came close to trained systems, though a feature-based trained model still led on several traits.
An unexpected result: MADRAG with GPT-5 on the "Ideas" criterion exceeded the agreement between human examiners (QWK 0.75 versus 0.69 on essay set 7; 0.67 versus 0.53 on set 8). In other words, on this one trait its scores matched the reference grades more closely than two human graders matched each other. These are single runs, without variance estimates.
Adaptations and Extrapolations
🔧 Adaptation: generate references before scoring
Don't have ready-made examples for your criteria? Ask the LLM to create them:
Before scoring, create three references for the criterion "{criterion}":
— Reference 2/10: a text that clearly fails on this criterion
— Reference 5/10: a text that passes but has no real strengths
— Reference 9/10: a text that wins convincingly on this criterion
Then score my text by comparing it to these references.
My text: {text}
The model first builds the anchors, then scores relative to them. This is our extrapolation: the paper used real scored essays, and self-made references may calibrate less well.
🔧 Adaptation: Advocate and Skeptic as a team before a decision
The role-splitting principle isn't just for scoring. Use it before important decisions:
I am making a decision: {decision}.
ADVOCATE: find only the arguments in favor. Don't mention risks.
[Advocate:]
SKEPTIC: find only the arguments against. Don't mention upsides.
[Skeptic:]
ARBITER: weigh both lists. What would a veteran angel investor
who has seen 500 startups decide, and why?
[Arbiter:]
Works for: choosing between job offers, launching a new product, changing your positioning. The Advocate defends the idea, the Skeptic tries to kill it, and the Arbiter makes the balanced call.
Resources
MADRAG: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring
Authors: Ali Keramati, Shiyuan Zhou, Sharad Mehrotra, Mark Warschauer Organization: University of California, Irvine Contact: [email protected]
ASAP dataset: https://www.kaggle.com/c/asap-aes/data
Related work cited in the paper: ChatEval (Chan et al., 2023), MAGIC (Jordán et al., 2025), CAFES (Su et al., 2025)