Criteria + Ensemble: How to Make LLM Ratings Reliable

STEP 1 (in one prompt):
  Standard criteria + {specific criterion, 1 sentence}
  → The model focuses on the dimension that matters

STEP 2 (same or separate requests):
  Ask for k=3–8 independent ratings
  → Average the results

PICKING THE WINNER:
  The option with the highest average score

Both steps work in a single prompt — you can ask the model to give 5 ratings in a row in one request.

Task: You wrote a description of your Substack newsletter for a promo post. You want to know how persuasive it is before you sink your budget into sponsored placements.

Prompt:

Rate this newsletter description on a scale of 1 to 10.

When rating, consider: persuasiveness, clarity of the value proposition, 
engagement level, specificity of the promise. Pay special attention 
to whether the reader understands WHAT exactly they'll get and HOW it's different 
from other newsletters in the niche.

[Description text:]
"An investing newsletter for people who are tired of the hype. I break down real 
cases, run the numbers, explain the risks. No promises of 10x returns — just honest 
analysis. 47,000 subscribers, new issues 3 times a week."

Give 5 independent ratings. Each time, rate as if for the first time, 
without looking back at the previous rating. Output all 5 numbers, then the average.

Result: The model will output 5 separate numerical ratings — each with or without a short rationale (depending on the prompt). At the end — the average. The spread between ratings shows how ambiguous the text is: if the spread is 4–7, different people read the text differently. If it's 6–7, it's stable. That's an insight in itself.

An LLM doesn't compute — it samples. When you ask for a rating, the model doesn't run a deterministic analysis. It generates the text of an answer, and that process involves randomness (temperature). One request — one random point. Several requests — several points, whose average is much closer to the "true" rating.

Generic criteria blur the focus. A standard "rate on a scale of 1–10" request leaves the model to decide what matters — usefulness, level of detail, creativity or something else. One clarifying sentence narrows the task and removes that ambiguity. It's not a complex instruction, just a clarification of priority.

Control levers: - k (number of ratings) — 3 is enough for a quick check, 5–8 for an important decision - The criterion sentence — change it to fit the task type: for text it's "persuasiveness," for code "logical correctness," for a business idea "realism of the assumptions" - Output format — you can ask for "numbers only, no explanations" (fast) or "number + one reason" (informative)

Rate {object} on a scale of 1 to 10.

When rating, consider: {standard criteria — for example: usefulness, clarity, 
persuasiveness, quality of arguments}. Pay special attention to 
{specific criterion — one concrete sentence for this task}.

{Object to rate}

Give {number} independent ratings in a row. Rate from scratch each time, 
as if seeing the text for the first time. Output each rating separately, 
and the arithmetic mean at the end.

What to fill in: - {object} — a text, an idea, an answer, a pitch, a description - {standard criteria} — general quality parameters for your task - {specific criterion} — one phrase about the key dimension (logic, substance, compliance with requirements, safety) - {number} — 3 for a quick check, 5–8 for important decisions

🚀 Quick start — paste into the chat:

Here is a template for reliable LLM-based rating. Adapt it to my task: 
[your task — for example: "rate a chapter of my book for persuasiveness"].
Ask me questions to fill in the template fields.

[paste the template above]

The LLM will ask what exactly to rate and which specific criterion matters — because without that the template works like a generic request, not a precise instrument.

⚠️ Subjective tasks: For ratings with no clear-cut answer ("how beautiful is this text"), ensembling reduces noise but doesn't solve the problem of the model's taste standing in for yours.

⚠️ Precise formatting requirements (Precise IF): This is the weakest category even after all the improvements — 48.8% versus 93% in the safety category. If you need to check compliance with exact technical requirements ("exactly 3 bullet points, each no longer than 10 words"), LLM rating is unreliable even with these techniques.

⚠️ Calibration via examples doesn't work: The idea of showing the model "here's a good example — here's a bad one" before rating sounds logical, but in practice it doesn't deliver a reliable gain. Don't waste effort hand-picking calibration examples.

⚠️ The cost of the ensemble: k=8 means 5x more requests. On free tiers with limits, you'll hit the caps fast. For most tasks, k=3–5 is a good balance.

The Composo AI team took 1,753 tasks from RewardBench 2 — a standard benchmark for evaluating "judge" models. Each task contains a user question + four candidate answers, one of which is pre-labeled as the best. The judge's job is to find it.

Baseline accuracy was measured simply: GPT-5.4 rates all four options once and picks the winner. It's correct 71.7% of the time. Then they added each technique separately and measured the gain.

Interestingly, calibration context (showing the model an example of a "good" or "bad" answer before rating) gave +1–2 pp only at k=1, and at k=8 it disappeared completely — the model had already stabilized through ensembling. This suggests that instability and lack of context are different problems, and the ensemble solves the first one better.

An unexpected result: a nano model with k=8 (the cheapest tier × 8 requests), at 0.4x the cost of a baseline request, reaches the same accuracy as the full model with a single request. That opens the door to cheap, reliable ratings.

An example of criterion injection for the "Math" category:

### Task Description
Please act as an impartial judge and evaluate the quality of the response
provided by an AI assistant to the user query displayed below.

Notes:
1- Your evaluation should consider factors such as the helpfulness, relevance,
   accuracy, depth, creativity, and level of detail of the response. Focus on
   whether the mathematical reasoning is logically valid, the steps are correct,
   and the final answer is accurate.
2- Begin your evaluation by providing a short explanation.
3- Be as objective as possible. After providing your explanation, please rate
   the response on a scale of 1 to 10.

[Query]
{prompt}

[Response]
{completion}

[Your judgement]

Context: This is the original judge prompt from the study. The sentence starting with "Focus on whether..." is the added criterion. Everything else is the standard template. One sentence, zero complexity, +12 pp of accuracy on math tasks.

💡 Adaptation: criteria for everyday tasks

Ready-made phrasings for specific criteria — just grab and paste:

For rating ad copy: "Pay special attention to whether there's a concrete and believable promise of results, and whether the text sounds like templated advertising."

For rating a resume: "Pay special attention to whether concrete work results are visible (numbers, facts) or just a list of responsibilities."

For rating a business idea: "Pay special attention to how realistic the demand assumptions are — whether wishful thinking is being passed off as obvious."

For rating a talk/pitch: "Pay special attention to whether the problem is clear before the solution is explained, and whether there's a concrete call to action."

🔧 Technique: spread as a quality signal

When you get 5 ratings, look not only at the average but also at the spread:

...Output all 5 ratings, the average and the range (maximum minus minimum). 
If the range is more than 2 points, briefly explain what in the text 
causes that ambiguity.

A spread of 1–2 points → the text reads predictably. A spread of 3+ points → something in the text will land differently with different "readers." That's not a rating bug — it's information about the text.

🔀 Extrapolation: ensembling for any subjective task

The "ask several times and average" principle works beyond rating. It applies anywhere a task has no single correct answer:

Give {number} independent options for {a headline / a name / a slogan / a phrasing}.
Generate each one from scratch, without relying on the previous options.
After all the options, pick the best one by the criterion {criterion} and explain why.

Instead of one random option, you get a space of solutions — and the model picks from it by your criterion.

Title: An Empirical Investigation of Practical LLM-as-a-Judge Improvement Techniques on RewardBench 2

Authors: Ryan Lail (Composo AI)

Code and data: https://github.com/composo-ai/llm-judge-criteria-ensembling

Benchmark: RewardBench 2 — Lambert et al., 2025

Key references from the paper: G-Eval (Liu et al.), MT-Bench (Zheng et al.), Prometheus 2, FrugalGPT (Chen et al.)