TL;DR
The study shows three simple techniques for evaluation and scoring tasks: (1) test semantically identical prompt variants — they produce different results, (2) average scores across different phrasings instead of repeating one prompt, (3) allow fractional answers even when the correct answers are whole numbers. Fractional scores (2.7, 3.2) act as gradations of the model's confidence.
The main finding: semantically identical prompts produce very different correlations with expert scores — from 0.176 to 0.279 on the same task. Swapping "evaluate" for "give a score" or "rank" changes the result. Counterintuitively, mentioning "based on this text" lowers quality — the model works better when you don't draw attention to the fact that it's seeing incomplete data. Allowing fractional scores widens the spread of answers (from 2-3 distinct values to 14-18), even though fractional scores are technically wrong — the correct answers are whole numbers only. But it's exactly this spread that helps rank items more accurately.
The technique works through three mechanisms: (1) testing variations finds the best phrasing for a specific model, (2) averaging different prompts lowers the chance of getting "no answer" and reveals the model's confidence through the spread of scores, (3) fractional scores free the model from rounding — instead of "is this a 2* or a 3?" it outputs 2.7, showing its real internal estimate.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
Three independent techniques (each can be used on its own):
TECHNIQUE 1: Prompt Perturbation (testing variations)
1. Take your base evaluation prompt
2. Create 3-5 semantically identical variants
(swap "evaluate" → "give a score", "rank", "assign a rating")
3. Run each variant
4. Compare the results → pick the best
TECHNIQUE 2: Fraction Facilitation (allowing fractions)
Add to the prompt: "or a value in between" / "fractional scores are allowed"
→ The model outputs 2.7*, 3.2* instead of only 2*, 3*, 4*
→ More gradations = more accurate ranking
TECHNIQUE 3: Multi-Prompt Averaging (averaging different prompts)
1. Run 3-5 different phrasings on the same task
2. Average the scores → you get a consensus score
3. The spread shows the model's confidence
All three techniques run in an ordinary chat as separate requests.
Example in Practice
Task: You're evaluating three startup ideas for an investor pitch. You need to pick the strongest one to present at Y Combinator Demo Day.
Ideas:
- A: A marketplace for renting construction equipment to small businesses
- B: An AI assistant for drafting cost estimates and bid documents in construction
- C: A service for finding subcontractors with a reputation system and risk insurance
Prompt 1 (base, with fractions):
Evaluate the innovativeness, demand, and feasibility of this business idea on a scale of 1 to 5; fractional scores are allowed:
[paste the description of idea A]
Prompt 2 (variation):
Give this idea a potential score from 1 to 5 (fractional values are fine):
[paste the description of idea A]
Prompt 3 (another variation):
Rank its appeal to an investor from 1 to 5 or a value in between:
[paste the description of idea A]
Repeat for ideas B and C.
Result: You get 9 scores (3 prompts × 3 ideas). For each idea, average the three scores from the different phrasings. For example: idea A got 3.2, 3.7, 3.5 → average 3.47; idea B: 4.1, 3.8, 4.2 → average 4.03; idea C: 2.8, 3.1, 2.9 → average 2.93. Ranking: B > A > C. The spread of scores shows confidence — a narrow spread for idea C (2.8-3.1) means the model is sure it's middling; a wide one for A (3.2-3.7) means there's uncertainty.
Why It Works
Why models are sensitive to phrasing: LLMs are trained on text where "evaluate the article" and "give the article a score" appear in different contexts — with different types of answers, different strictness, different formats. The model activates different response patterns depending on the exact wording. It's not a bug — it's a feature of the statistical nature of language models. Every word shifts the probabilities of the next tokens.
Why fractional scores help: Internally, the model operates on probabilities, not categories. When you ask it to choose "2* or 3," the model rounds an internal value of 2.7. When you allow fractions, it outputs 2.7 directly. That is NOT a more accurate answer to "which score is correct?" but a more informative one — it reveals internal confidence. An article scored 2.9 is closer to 3 than an article scored 2.3, even though both are formally "between 2* and 3*".
Why averaging different prompts works: Each phrasing activates slightly different aspects of the model's knowledge. "Evaluate innovativeness" focuses on novelty, "investor potential" on business metrics, "appeal" on overall impression. Averaging gives a more balanced score. Plus, it lowers the risk of "the model misunderstood and wrote an essay instead of a score" — if one prompt fails, the others still give scores.
The counterintuitive finding: The phrase "based on this text" or "based on the title and abstract" lowers score quality. A likely reason: the model starts doubting whether the data is sufficient and gives more cautious (averaged-out) scores. Without the phrase the model simply works with what it has, without reflecting on its limitations.
Control knobs:
- Numeric scale range — shrink it to 1-3 for rough sorting (faster), widen it to 1-10 for fine gradation (but models tend to gravitate to the middle of the scale)
- Context explicitness — drop "based on this text" if the model gets too cautious
- How fractions are allowed — "fractional scores are allowed" gives more spread than "1, 1.5, 2, 2.5, 3..." (the latter yields only halves)
- Number of variations — 3 prompts for a quick check, 5-8 for critical decisions (a tokens/accuracy trade-off)
Prompt Template
Base template with fractional scores:
Evaluate {aspect} on a scale from {minimum} to {maximum}; fractional values are allowed:
{item_to_evaluate}
A set of 3 variations for averaging:
PROMPT 1:
Evaluate {aspect} on a scale from {minimum} to {maximum}; fractional values are allowed:
{item_to_evaluate}
PROMPT 2:
Give a {aspect} score from {minimum} to {maximum} (fractions are fine):
{item_to_evaluate}
PROMPT 3:
Rank {aspect} from {minimum} to {maximum} or a value in between:
{item_to_evaluate}
What to plug in:
- {aspect} — what you're evaluating: "text quality," "appeal to investors," "fit with requirements"
- {minimum} / {maximum} — the scale bounds, usually 1-5 or 1-10
- {item_to_evaluate} — the text to evaluate: a résumé, a business idea, a task description
Run each prompt as a separate message. Average the numeric results by hand or ask the model:
Here are three scores for the same item from different phrasings: {score1}, {score2}, {score3}.
Average them and explain what the spread shows.
🚀 Quick start — paste into the chat:
Adapt these three prompts to my evaluation task: {your task}.
Ask me which aspects to evaluate and what scale I need.
[paste the set of 3 variations above]
The model will ask what exactly you're evaluating, which scale (1-5, 1-10), and which criteria matter. It will take the "three semantically different phrasings + fractional scores" pattern and fit it to your task.
Limitations
⚠️ Different models — different best prompts: What worked for Gemma 27b barely works for ChatGPT 4o-mini (the prompt correlations are nearly opposite). There's no universal "best prompt" — you have to test on your specific model. Llama 4 Scout and Magistral showed weak agreement with Gemma; for Qwen 32b and DeepSeek R1 32b there was almost no difference between prompts.
⚠️ Takes several requests: The techniques rely on multiple iterations — at least 3 requests for averaging, 5-8 for a reliable result. That's tokens and time. For one-off evaluations it may be overkill; for critical decisions (picking from 100 candidates, choosing a strategy) it's worth it.
⚠️ Specific to ranking/evaluation tasks: The principles work when you need to compare and rank items, not necessarily predict an exact score. If you need a precise score (e.g., "percent match with requirements" for automated filtering), the technique doesn't fit, because the absolute values are inaccurate — the value is only in the relative order.
⚠️ Not for creative tasks: Prompt variations for text generation produce different content (that's a feature, not a bug). Here, prompt variations are used to stabilize a score through averaging. For creative generation, averaging is meaningless — you can't "average" two different texts.
How They Tested It
The team took 2,780 research articles from the UK's REF2021 research assessment with expert scores on a 1-4 scale (nationally recognised to world-leading). They created 58 prompt variations — changing the wording ("evaluate" vs "give a score" vs "rank"), adding/removing a mention of context ("based on the title and abstract"), and testing different ways of stating the scale ("1* 2* 3* 4" vs "from 1 to 4" vs "1 2* 3* 4* or a fractional value"). Each prompt was run 10 times through Gemma 27b — 1.7 million requests in total. They measured the Spearman correlation between the model's predictions and the expert scores.
Why the results came out this way: Correlations for semantically identical prompts ranged from 0.176 to 0.279 — a 1.5× difference. That's a huge gap for the exact same task. The explanation: models are trained on billions of texts where "evaluate the article" and "give the article a score" appeared in different contexts — different answer formats, different strictness. Every word shifts the probability distribution of the next tokens.
The surprise: Prompts with the phrase "based on the title and abstract" lowered the correlation (0.17-0.23) compared with prompts without it (0.33-0.47). The likely reason: explicitly pointing out incomplete data makes the model more cautious, giving more averaged-out scores and avoiding extremes. Without the phrase, the model just works with what it sees.
Fractional scores: Prompts with "fractional values are allowed" produced 14-18 distinct values instead of 2-3 for ordinary prompts (usually only 2, 3 and rarely 4). And the fractional prompts gave higher correlations (0.36-0.39 vs 0.33-0.37), even though the correct answers are whole numbers only. The paradox: allowing "wrong" answers improved the ranking. The interpretation: fractional scores reveal the model's internal confidence — 2.7 vs 2.3* are both "between 2* and 3," but the first is closer to 3.
Averaging different prompts: In 11 of 15 tests, averaging the scores from different semantically equivalent prompts gave a higher correlation than the best single prompt in the set. Plus, there were fewer cases where the model failed to give a score (wrote text instead of a number).
Checking other models: ChatGPT 4o-mini showed an almost opposite picture — prompts that succeeded for Gemma flopped for ChatGPT, and vice versa (the Pearson correlation between prompt ranks was negative). Llama 4 Scout and Magistral — weak agreement. Qwen 32b and DeepSeek R1 32b — almost no difference between prompts, they all worked about the same. Insight: There's no universal "best prompt" for every model, but the strategy of averaging different phrasings worked for all models — a safe bet when you don't know which prompt is best.
Adaptations and Extrapolations
Technique 1: Explicit uncertainty request
Instead of allowing fractional scores, ask the model to state its confidence explicitly:
Evaluate {item} on a scale from 1 to 5.
State your confidence as a percentage and the range of possible scores.
Answer format:
Score: [number]
Confidence: [percent]
Range: [min-max]
Effect: You get explicit meta-information instead of implicit (through fractional scores). "Score: 3, Confidence: 60%, Range: 2-4" is more informative than just "3" or "3.2".
Technique 2: Pairwise perturbation (variations for pairwise comparison)
If you need to pick the better of two options, use several phrasings of the question:
PROMPT 1: Which option is better — A or B?
PROMPT 2: If you had to pick only one — A or B?
PROMPT 3: Which option will reach the goal more effectively — A or B?
Effect: If all three prompts chose A — high confidence. If 2:1 — moderate. If the answers differ — you need a deeper analysis or a third option.
Technique 3: Multi-aspect perturbation (variations by aspect)
For a comprehensive evaluation, vary not the phrasing but the aspect being scored:
PROMPT 1: Evaluate the idea's innovativeness from 1 to 5.
PROMPT 2: Evaluate the idea's feasibility from 1 to 5.
PROMPT 3: Evaluate the idea's commercial potential from 1 to 5.
PROMPT 4: Evaluate its appeal to investors from 1 to 5.
Effect: Four different scores show the item's profile. Averaging gives an overall score, but the spread matters more — an idea can be super innovative (5) yet unfeasible (2), valuable information that a single "evaluate the idea overall" prompt would lose.
Technique 4: Temperature perturbation (if you have access to parameters)
For models accessed via API, combine prompt variations with temperature variations:
One prompt × 3 temperatures (0.3, 0.7, 1.0) = 3 different scores
3 prompts × 3 temperatures = 9 scores to average
Effect: Temperature changes the model's "creativity." Low (0.3) gives conservative scores, high (1.0) more varied ones. Averaging balances "what the model considers most likely" against "what alternatives are possible."
Resources
Prompt perturbation and fraction facilitation sometimes strengthen Large Language Model scores / Mike Thelwall / University of Wolverhampton, UK