TL;DR
LLMs grade with a pull toward the middle: ask one to rate a text on a 1-to-10 scale and you'll almost always get a 6 or 7. That's no coincidence: the model avoids extreme judgments and drags scores toward the center of the scale. As a result, "good" and "excellent" work get the same score.
The reason is that the model has no external anchor. It produces a score from general impressions rather than by comparing against concrete examples of strong and weak work. On top of that, a single agent hunts for pros and cons at the same time, they cancel each other out, and the verdict slides to "fine."
MADRAG fixes this in two moves: it splits the search for strengths and weaknesses across separate roles (Advocate → Skeptic → Judge), and it gives the Judge examples of work at different quality levels. The Advocate talks only about strengths. The Skeptic, only about weaknesses. The Judge sees both views plus the reference examples and delivers a calibrated score.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: ADVOCATE: strengths only → a written argument (no score)
STEP 2: SKEPTIC: weaknesses only, taking the Advocate's argument into account →
a written rebuttal (no score)
STEP 3: JUDGE: reads both arguments + reference examples at different levels →
final score with justification
All three steps fit in one prompt (or three separate messages)
Example
The task: Jake Turner has written an investor pitch for his B2B SaaS product. He wants to know how convincing the pitch really is, not just "yeah, it's fine." The usual "rate my pitch" request returns 7/10 almost every time.
Prompt:
You are an expert pitch evaluation system. Play three roles in sequence.
Here is the pitch:
"[paste the pitch text]"
Evaluation criteria: persuasiveness for an investor (clarity of the problem,
market size, uniqueness of the solution, team, traction).
Scale: 1–10.
---
ROLE 1: ADVOCATE
Find only the strengths of the pitch.
Don't mention weaknesses. Don't give a score. Only arguments in favor.
[Advocate:]
---
ROLE 2: SKEPTIC
Read the Advocate's arguments and the pitch itself.
Find only the weaknesses: what isn't convincing, what raises doubts,
what's missing.
Don't mention strengths. Don't give a score. Only arguments against.
[Skeptic:]
---
ROLE 3: JUDGE
Read both arguments. Use the reference examples below for calibration.
Reference 1 (score 2/10): "We're building an app for everyone. The market is huge.
We have no competitors."
Reference 2 (score 5/10): "We solve reporting automation
for small businesses. The market is $5B. There are 3 competitors; we're faster.
No customers yet; we launch next quarter."
Reference 3 (score 9/10): "200 paying customers, $120K MRR,
15% month-over-month growth. We solve a CFO pain point at companies with 50–500 employees.
Competitors solve an adjacent problem, not this one. Team: ex-CFO from Stripe + ex-CTO from Intuit."
Compare the pitch to the references. Give a score on the 1–10 scale
with a concrete justification: why not higher and why not lower.
[Judge:]
Result: The model will produce three blocks in sequence. The Advocate lists concrete strengths. The Skeptic lists pinpoint weaknesses, building on the Advocate's arguments. The Judge gives a score positioned "relative to the references", for example: "6/10: strong team and a clear problem, but no traction and a fuzzy ICP. Closer to reference 5, slightly higher for team quality." Not "fine", but a specific position on the scale.
Why It Works
The problem: an LLM grading head-on sees pros and cons at the same time, and they get averaged right in the middle of generation. The model doesn't "weigh" anything; it just generates the next token, and the balance of arguments literally pulls the answer toward the center of the scale.
What models do well: play roles with hard constraints. Tell it "strengths only, no weaknesses" and the model faithfully stays inside the frame. It can argue within set boundaries instead of just generalizing.
How the method exploits this: splitting into Advocate and Skeptic forces the viewpoints apart. Each role generates a deliberately one-sided but honest argument. The Judge receives views that are already separated instead of a mush, which makes it easier to deliver a weighted judgment rather than an average one. References at different points of the scale provide an anchor: the model compares instead of guessing an absolute score.
Control knobs:
- Number of references → at least 3 (low / medium / high). You can add 5 for finer calibration
- Role strictness → tighten the ban: "Advocate: if you mention a weakness, rewrite" holds the frame even harder
- Criteria → the Judge scores one criterion more accurately than five at once. Split into separate requests per criterion
- Judge output format → add "explain why not 1 point higher and not 1 point lower" to pull out concrete boundary reasoning
Prompt Template
You are a system for evaluating {what we are evaluating}.
Here is the material to evaluate:
"{text}"
Criteria: {evaluation criteria}.
Scale: {minimum}–{maximum}.
---
ROLE 1: ADVOCATE
Strengths only. No scores. No weaknesses.
[Advocate:]
---
ROLE 2: SKEPTIC
Weaknesses only, taking the Advocate's arguments into account.
No scores. No strengths.
[Skeptic:]
---
ROLE 3: JUDGE
Read both arguments. Use the references for calibration:
Reference for {low_score}: "{example of weak work}"
Reference for {mid_score}: "{example of average work}"
Reference for {high_score}: "{example of strong work}"
Compare against the references. Give the final score with a justification:
why not 1 point higher and why not 1 point lower.
[Judge:]
What to fill in:
- {what we are evaluating}: a pitch, a sales proposal, an article, a script
- {evaluation criteria}: specific ones, such as "persuasiveness for the client", "clarity of structure", "concreteness of arguments"
- {example of weak/average/strong work}: real examples. Use actual texts you've seen, or ask the LLM to generate references before running the method
🚀 Quick start: paste this into the chat:
Here is the MADRAG template for evaluating texts.
Adapt it to my task: {your task}.
Ask me questions to fill in all the fields.
[paste the template above]
The LLM will ask what to evaluate, by which criteria, and for reference examples, because without anchors on the scale the Judge can't calibrate the score. It will take the template's structure and assemble a working prompt for your task.
Limitations
⚠️ Weak on countable criteria: grammar, spelling, number of errors: debate doesn't help here. The Advocate and Skeptic don't count typos any more accurately than one attentive agent. For criteria like these, a direct "find all the errors" prompt works better.
⚠️ Weaker without references: if you don't give the Judge examples at different levels of the scale, half the method's power is gone. The study showed that the reference examples are the main driver of calibration. Debate adds 4–5%, but references add 20–30% accuracy.
⚠️ Subjective taste criteria: "I like the style" without clear criteria: the method won't help. You need observable features you can put into words.
⚠️ Prompt length: three roles plus references make for a big context. For very short tasks (rate a single paragraph) a direct prompt is simpler.
How They Tested It
A team from the University of California, Irvine took the standard ASAP dataset of student essays (thousands of texts graded by human examiners) and asked a tough question: can a system with no training get close to the accuracy of trained neural networks?
They tested MADRAG on GPT-3.5, GPT-4o and GPT-5, with identical prompts for all models. They compared it against two kinds of systems: ones trained on the dataset (ProTACT, FeatEng-RF) and other training-free prompting approaches (ZS-LLM, CSR-J). The metric was Quadratic Weighted Kappa: roughly, how closely the system's score matches a human examiner's.
The most interesting part surfaced in the ablation tests (checking "what happens if we remove one component"). It turned out that debate and references do different jobs. References fix the pull toward average scores: the system stops being afraid to give a 2 or a 9. Debate improves reasoning about content criteria (ideas, argument structure) but barely helps with mechanical ones (grammar, punctuation).
An unexpected result: MADRAG with GPT-5 on the "Ideas" criterion exceeded the agreement between human examiners (0.75 versus 0.69 for humans). In other words, the system turned out to be _more consistent_ than people at scoring the substance.
Adaptations and Extrapolations
🔧 Adaptation: generate references before scoring
Don't have ready-made examples for your criteria? Ask the LLM to create them:
Before scoring, create three references for the criterion "{criterion}":
— Reference 2/10: a text that clearly fails on this criterion
— Reference 5/10: a text that passes but has no real strengths
— Reference 9/10: a text that wins convincingly on this criterion
Then score my text by comparing it to these references.
My text: {text}
The model first builds the anchors, then scores relative to them: calibration without pre-made examples.
🔧 Adaptation: Advocate and Skeptic as a team before a decision
The role-splitting principle isn't just for scoring. Use it before important decisions:
I am making a decision: {decision}.
ADVOCATE: find only the arguments in favor. Don't mention risks.
[Advocate:]
SKEPTIC: find only the arguments against. Don't mention upsides.
[Skeptic:]
ARBITER: weigh both lists. What would a veteran angel investor
who has seen 500 startups decide, and why?
[Arbiter:]
Works for: choosing between job offers, launching a new product, changing your positioning. The Advocate defends the idea, the Skeptic tries to kill it, and the Arbiter makes the balanced call.
Resources
MADRAG: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring
Authors: Ali Keramati, Shiyuan Zhou, Sharad Mehrotra, Mark Warschauer Organization: University of California, Irvine Contact: [email protected]
ASAP dataset: https://www.kaggle.com/c/asap-aes/data
Related work cited in the paper: ChatEval (Chan et al., 2023), MAGIC (Jordán et al., 2025), CAFES (Su et al., 2025)