TL;DR
When you ask an LLM to grade something (a text, an idea, an answer), the choice of grading scale affects the result. The study compared three scales: 0-5, 0-10 and 0-100 — on six types of tasks, from objective (text toxicity) to subjective (answer quality). 12 people and 6 different models graded the same examples on all three scales.
On subjective tasks (judging text quality, how useful an answer is) models lose consistency — both with each other and with humans. And the different scales produce different levels of agreement with human judgment: the 0-5 scale shows the highest match, 0-10 the weakest, even when agreement within each group (humans on their own, models on their own) is high. This isn't a linear rescaling of points — models interpret the task differently depending on the scale.
The finding yields a concrete recommendation: use a 0-5 scale for grading tasks, especially subjective ones. It doesn't guarantee perfect agreement with your own opinion, but statistically it gives the best alignment with how humans grade.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Example
Task: You're evaluating three versions of a cold email to a prospect for a new task-management SaaS. You need to figure out which version sounds more persuasive and makes people want to reply.
Prompt:
Rate three versions of a cold email on these criteria:
- Persuasiveness (how well it hooks into the customer's pain)
- Reply appeal (does it make you want to continue the conversation)
- Professionalism (does it read like spam)
Use a 0-5 scale for each criterion, where:
0 = total failure
5 = perfect
Version 1:
[text of email 1]
Version 2:
[text of email 2]
Version 3:
[text of email 3]
Give the scores as a table, then explain the main strengths and weaknesses of each version.
Result:
The model will produce a table with scores on the three criteria for each email, followed by brief justifications. Scores on a 0-5 scale will be closer to how a human would grade than if you'd asked for a 0-10 or 0-100 scale. Especially on subjective criteria like "reply appeal" — where there's no clear rule, only a gut feeling.
Why It Works
The problem: an LLM calibrates scores differently depending on the scale. On objective tasks (say, "how toxic is this text") the scale barely matters — the criteria are clear and the model stays consistent. But on subjective tasks (text quality, answer usefulness) models start interpreting the task differently: what does "7 out of 10" mean vs "70 out of 100"? Even if mathematically they're the same, the training patterns differ — models have seen 1-5 ratings (reviews, star ratings) far more often than 0-100.
People change their behavior too depending on the scale. A 0-100 scale creates an illusion of precision — it feels like you can tell 68 from 72, though in practice that's noise. A 0-10 scale brings out "edge avoidance" — people rarely give 0 or 10 and cluster in the middle. The 0-5 scale is the psychological sweet spot: enough gradations to distinguish levels, but not so many that calibration gets confusing.
Result: The 0-5 scale minimizes the mismatch between how the model calibrates scores and how people do. On 0-10, models more often "smear" points in a way that feels unnatural to humans. On 0-100, you get false granularity — the model outputs 73 even though it has no such precision inside.
Bonus: If you want to see the reasoning, add before the score: "First explain your observations, then give the score." This triggers Chain-of-Thought and can improve calibration, especially on complex criteria.
If the task is objective (toxicity, factual consistency) — the scale barely matters, use any. If the task is subjective (quality, persuasiveness, style) — 0-5 gives the highest agreement with how people grade.
Prompt Template
Rate {object being graded} on the following criteria:
1. {Criterion 1} — {what we're assessing}
2. {Criterion 2} — {what we're assessing}
3. {Criterion 3} — {what we're assessing}
Use a 0-5 scale, where:
0 = {worst case}
5 = {ideal case}
{The object to grade, or several versions}
Output format:
- A table with scores for each criterion
- A brief justification for each score
What to fill in:
- {object being graded} — a text, idea, answer, design, strategy
- {criterion} — the specific aspects that matter (clarity, persuasiveness, usefulness)
- {worst case} / {ideal case} — scale anchors so the model understands the boundaries
Example of concrete anchors: - For "persuasiveness": 0 = sounds like spam, 5 = impossible not to be intrigued - For "clarity": 0 = makes no sense at all, 5 = crystal clear on the first read
Limitations
⚠️ Objective tasks: If the task has clear criteria (text toxicity, factual accuracy), the difference between scales is minimal. The advantage of 0-5 shows up on subjective judgments.
⚠️ Specific benchmark: On MT-Bench (judging the quality of conversational answers), the 0-10 scale showed better agreement than 0-5. If you regularly work with one particular type of task, test the scales on your own examples — the averaged result may not hold for you.
⚠️ Individual models: GPT-4o and Gemini showed the best agreement with humans on all scales. Smaller models (Llama, Qwen, Mistral) lose calibration more. If you're using a weaker model, the gain from 0-5 may be smaller.
⚠️ Gender differences: In the study, women aligned better with the LLMs on the 0-10 scale, men on 0-5. If you're grading for a specific audience, keep in mind that calibration can shift.
How It Was Studied
The team assembled 12 graduate students (6 women, 6 men) and 6 models (GPT-4o, Gemini, Llama, Qwen, DeepSeek, Mistral). Each person and each model graded the same examples on three scales: 0-5, 0-10 and 0-100. In total, 5497 examples from six benchmarks — from objective (text toxicity, sentence similarity) to subjective (answer quality in a dialogue, grading resumes).
They used ICC (intraclass correlation coefficient) — a psychometrics metric that measures absolute agreement, not just correlation. Where Pearson shows "do they rank things the same way", ICC shows "do they give the same scores". That's critical for grading — what matters is not only that A is better than B, but by how much.
The main result: On objective tasks (STS-B — sentence similarity, ToxiGen — toxicity) all scales work equally well, with ICC between models of 0.94-0.95. But on subjective ones (MT-Bench — dialogue quality, SummEval — summary quality) agreement between models drops to 0.57-0.63 on the 0-5 scale, and even further on the other scales. Meanwhile, humans stay consistent (ICC 0.93-0.95) even on subjective tasks.
Comparing humans and LLMs, they found: the 0-5 scale gives an ICC of 0.853 between the averaged human panel and the averaged model panel. The 0-10 scale — only 0.805, the worst result. That's counterintuitive: you'd think more gradations = more precision. But more gradations = more calibration confusion on subjective tasks.
A surprising finding: They ran the models at different temperatures (0.1, 0.4, 0.7, 1.0) — agreement barely changed. So it's not about generation randomness, it's about how the model interprets the scale itself when parsing the task.
The gender angle: Women in the experiment showed better agreement with the LLMs on the 0-10 scale (ICC 0.844), men on 0-5 (ICC 0.870). Both groups were most internally consistent on 0-5. The reasons are unclear — perhaps different groups have different score-calibration patterns.
Resources
Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale Weiyue Li, Minda Zhao, Weixuan Dong, Jiahui Cai, Yuze Wei, Michael Pocress, Yi Li, Wanyan Yuan, Xiaoyue Wang, Ruoyu Hou, Kaiyuan Lou, Wenqi Zeng, Yutong Yang, Yilun Du, Mengyu Wang Harvard University, CMU, Stanford University, UC San Diego