TL;DR
When you ask an LLM to score texts (sentiment, toxicity, political lean), the scale in the prompt itself shapes the result. Change the format (decimals or whole numbers), the range (3, 5, 11, 21 points) or the sign (−5 to +5 versus 0 to 10), and the scores shift. The same text gets a different score, and sometimes a different category.
On average, changing a single element of the scale moves the score by roughly a tenth of the scale. And about one text in five jumps to a different category: "neutral" turns "toxic," "left" turns "centrist." Toxicity takes the hardest hit, with 29–40% of texts switching category. A detailed rubric in the prompt (breaking down criteria before the final score) doesn't help: stability is the same as with a plain "rate this."
The practical takeaway: scores from different scales can't be compared or merged. Pick your scale deliberately, to fit what you're measuring, and lock it in once. Check important conclusions against a second, alternative scale.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
This isn't a new prompt, it's a set of rules for handling the scale that the authors recommend:
STEP 1: Identify the trait type → bipolar (good↔bad, left↔right)
or unipolar (present/absent and how much: toxicity, spam, aggression)
STEP 2: Pick ONE scale for that type → lock in the format, range, and sign
STEP 3: Run a sample (in separate requests) on a second, alternative scale
STEP 4: Compare: which texts changed category → that's the "gray zone," review it by hand
STEP 5: Check against human labels on 30–50 examples, if you have them
Steps 3–4 run in separate requests: don't mix two scales in one chat.
Example in Action
Task: You run a YouTube channel with 200,000 subscribers. Every video gets hundreds of comments, and you want to auto-flag the toxic ones so moderators see them first. The threshold: "anything above 7 out of 10 goes to manual review."
Here the scale is everything: toxicity is exactly where changing it breaks the results the most.
Prompt (request A):
You are rating the toxicity of comments on a YouTube channel about business.
Toxicity is a trait that runs from "none" to "extreme": 0 means no toxicity
at all, 10 means maximum toxicity (insults, harassment, threats).
Rate each comment with a whole number from 0 to 10.
Return only JSON: [{"id": 1, "toxicity": N}, ...]
Comments:
1. "Did you even read what you wrote? Total nonsense"
2. "People like you should be run out of the market, you hack"
3. "Disagree with your Amazon numbers, the margins there are lower"
4. "Oh sure, yet another guru selling courses"
Prompt (request B, in a new chat, same comments):
Rate each comment on a toxicity scale from 1 to 5, where
1 = non-toxic, 3 = moderately toxic, 5 = extremely toxic.
Return only JSON: [{"id": 1, "toxicity": N}, ...]
Result: You'll get two JSON lists. Convert both to a fraction of the scale (0–1) and compare. The clearly harmless comment (#3) and the clearly rude one (#2) will most likely match. The borderline ones (#1, #4, with sarcasm and minor rudeness) may land in different categories. Those are the ones to hand to a moderator instead of trusting the "above 7" threshold.
Why It Works
The LLM's weak spot. The model doesn't measure toxicity with a ruler. It picks a number that "sounds right" for the given scale. Numbers aren't firmly tied to meaning: "3 out of 5" and "6 out of 11" are different answers to it, even though as fractions they're nearly the same. So the format, length, and sign of the scale change which number pops out.
The quirk of unipolar traits. Toxicity runs from "none" to "very." A −5 to +5 scale puts negative numbers on "no toxicity," and the meaning of the bottom end gets fuzzy. Humans behave similarly: on scales with negative numbers they pick the lower half less often. The authors are careful to note that they had only one unipolar task, so other explanations can't be ruled out.
Why the rubric doesn't save you. A rubric clarifies what to rate. But the problem is elsewhere: how the model translates its judgment into a number. Criteria don't help here, because the ambiguity lives in the number itself.
Control levers: - Match scale type to trait type → for unipolar traits, use an unsigned scale starting at 0. - Format → if you use whole numbers, remember that the range then matters more (ordinal scales amplify range changes). Decimals react more strongly to a sign change. - Scale length → the longer the scale, the smaller the numeric shift, but the share of texts switching category stays the same. More points doesn't mean more stable. - A second scale → any independent scale gives you a robustness check.
Prompt Template
You are rating {trait} in {text_type}.
Definition: {trait} is {unipolar: from "none" to "extreme" /
bipolar: from "{pole_1}" to "{pole_2}"}.
Scale: {format: whole number / number with one decimal} from {min} to {max}.
{min} means {what_min_means}, {max} means {what_max_means}{, the midpoint means what_mid_means}.
Return only JSON: [{"id": N, "{field_name}": number}, ...]
Texts:
{numbered_list_of_texts}
What to fill in:
- {trait} — toxicity, sentiment, aggression, formality, and so on.
- {min}/{max} — for a unipolar trait use 0…10 or 1…5, no negatives.
- For a bipolar trait, either a signed scale (−2…+2) or an unsigned one (1…5) works. What matters: the same one in every run.
- In the second (check) run, change one thing — format, range, or sign. Leave everything else alone.
Limitations
⚠️ No ground truth: the paper shows that scores are unstable, but doesn't show which scale is more accurate (there's no comparison against human labels). The "unsigned for toxicity" recommendation rests on the authors' reasoning, not a direct comparison.
⚠️ One unipolar task: the toxicity conclusion rests on a single trait. Other unipolar traits (spam, aggression) are probably similar, but that's not proven.
⚠️ Final score only, and low-effort model modes: all models ran at the minimum reasoning level. How much the results would change with "thinking" modes, the paper doesn't say.
⚠️ Coarse categories: category changes were counted using three equal thirds of the scale. For your own threshold (say, "above 7") the numbers will differ, but the direction holds.
⚠️ A rubric isn't a cure-all, but it doesn't hurt either: it may improve other qualities of the scores (consistency); here only robustness to scale changes was tested.
How They Tested It
The idea is simple: score the same text on different scales, changing only one element at a time. The authors crossed format (decimals and whole numbers), range (3, 5, 11, 21 points) and sign (with and without negatives). That produced 16 scales. They tested them on three tasks (political bias, review sentiment, toxicity), two prompt styles (a plain "rate this" and a detailed rubric) and five models. Texts came from labeled datasets: 1,700 news sentences and 5,000 each of reviews and comments.
To compare numbers, all scores were normalized to a 0-to-1 range. Then they compared only "minimal pairs": two scales that differ by exactly one element (40 pairs total). They measured two things: how far the score moved and whether the text changed category (low, medium, high).
The result was strikingly uniform. Changing the format, range, or sign produced a shift of about 0.1 and a category change for 18–20% of texts. The biggest surprise: the rubric didn't help: the difference from the plain prompt never exceeded 0.007. All five models reacted the same way, and the strongest factor turned out to be the task: for toxicity 29–40% of texts changed category, for political bias 13–15%, for sentiment 8–10%. There are subtler effects too. Ordinal scales react more strongly to a range change (21.5% vs. 14.9%). Decimal scales react more strongly to a sign change (24.4% vs. 16.3%). The practical takeaway: the scale and the task shape the result together, and there's no universally safe scale.
Adaptations and Extrapolations
🔧 Technique: threshold → "gray zone" instead of a hard cutoff
Instead of a single "above 7 = toxic" threshold, do two runs and send everything where the categories disagree to manual review:
Category on scale A: low / medium / high
Category on scale B: low / medium / high
If they don't match → "send to manual review"
This is my adaptation of the authors' logic. They measured the disagreement; they didn't propose this rule.
Extrapolation: checking an LLM judge. If you rate the quality of texts or agent responses with a 1–10 score (LLM-as-a-judge), the same principle applies: rerun the scoring on a second scale (say, 1–5). If the leader of the list changes, your conclusion "version A beats version B" is resting on the scale, not on the text. This is a transfer of the principle; the paper doesn't cover this task.
Resources
- Paper: Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring
- Authors: Wonjeong Jo (UC Davis), Gyuwan Kim (UC Santa Barbara), Ming Shan Hee (MBZUAI)
- Data: MBIC (Media Bias Annotation Dataset), VADER movie reviews, Jigsaw Unintended Bias in Toxicity Classification
- From psychometrics: Schwarz et al., 1991 and Tourangeau et al., 2007 — on how the sign of a scale affects human responses