Reflect-and-Revise: Iteratively Improving Scoring Criteria by Reflecting on Mistakes

ITERATION (repeat 3-10 times):

STEP 1: Score with the current rubric
→ The model scores 5-10 examples
→ For each: score + rationale

STEP 2: Collect the discrepancies
→ Compare the model's scores to the reference scores
→ Identify error patterns

STEP 3: Improve the rubric
→ The model analyzes its mistakes
→ Rewrites the criteria
→ Adds concrete indicators

STEP 4: Validation (optional)
→ Test the new rubric on held-out examples
→ Keep the best version

All steps happen in a conversation with the model; no code or API required.

Task: You're a product manager at an accelerator scoring startup pitches. You need to give 1-5 scores on four criteria: problem, solution, market, team. But the scores come out inconsistent: similar pitches get a 3 one time and a 4 the next.

Iteration 1: A simple rubric:

Score this startup pitch on a 1-5 scale, considering:
- Clarity of the problem
- Uniqueness of the solution  
- Market size
- Team experience

[Paste the pitch]

Give a score and a rationale.

You score 5 pitches and record the model's scores alongside your reference scores.

Iteration 2: Error analysis:

Here are my scores for 5 pitches and yours:

Pitch 1: I gave 4, you gave 3: you underrated the team
Pitch 2: I gave 2, you gave 4: you overrated a vague problem
Pitch 3: I gave 5, you gave 5: match
Pitch 4: I gave 3, you gave 2: you underrated a niche market
Pitch 5: I gave 3, you gave 4: you overrated a solution with no evidence

Rewrite the rubric to eliminate these errors. 
Add concrete indicators for each score.

Iteration 3: The refined rubric:

The model will produce something like:

PROBLEM (1-5):
5: A concrete figure for the loss/pain + statistics
4: A clear statement without numbers
3: A general but understandable statement
2: Vague, not tied to an audience
1: Not stated

TEAM (1-5):
5: Relevant experience >3 years + a product track record
4: Relevant experience 1-3 years
3: Experience in an adjacent field
2: No relevant experience
1: No information

[and so on for every criterion]

Repeat 1-2 more times on new examples until the scores stabilize.

Result:

The model will show the evolution of the rubric, from generic wording to concrete thresholds, numeric indicators and conditional rules. The final version will contain visual emphasis (key criteria highlighted), mapping tables (score → signs) and explicit rules ("if the team has no relevant experience → max 3 points").

LLMs are bad at keeping implicit criteria in mind. When you write "score the quality", the model interprets it differently every time. It considers the same signals one time and ignores them the next. There's no systematic checklist, just intuition, and intuition is unstable.

An LLM's strength is pattern matching on examples. The model is excellent at finding regularities in a series of "input → output" pairs. When it sees 5-10 cases where it got it wrong, it can derive a rule: "If the problem has no numbers, max 3 points". It's the same idea as few-shot learning, but applied to the scoring criteria themselves.

The method turns a fuzzy instruction into an explicit algorithm. After a few iterations the rubric gains:

• Numeric thresholds ("experience >3 years" vs "experience 1-3 years")
• Mandatory signals ("if there are no statistics → no higher than 4")
• Mapping tables (score → list of indicators)

That removes ambiguity: the model stops guessing and starts counting hits against the criteria.

Control knobs:

• Number of iterations (3-10): more iterations → a more detailed rubric, but gains slow down after iteration 5-7
• Batch size (5-20 examples): more examples → error patterns are easier to see, but processing takes longer
• Rationale format: ask the model to explain each score in detail → you see more precisely where it went wrong
• Starting rubric: you can start with the barest "Score 1-5" or with a ready detailed one; the results are comparable

Iteration 1: Initial scoring

Here are the scoring criteria:

{initial_rubric}

Score these {N} examples and give a rationale for each:

EXAMPLE 1:
{example_1}

EXAMPLE 2:
{example_2}

[...]

Answer format for each:
- Score: [points]
- Rationale: [why this score]

Iteration 2+: Improving the rubric

Here are the results of scoring {N} examples:

EXAMPLE 1:
My score: {reference_1}
Your score: {model_1}
Model's rationale: {rationale_1}
Where it went wrong: {error_analysis_1}

EXAMPLE 2:
My score: {reference_2}
Your score: {model_2}
Model's rationale: {rationale_2}
Where it went wrong: {error_analysis_2}

[...]

Current rubric:
{current_rubric}

Analyze the patterns in the discrepancies and improve the rubric:
1. Add concrete indicators for each score
2. Specify thresholds where possible
3. Write explicit rules ("if X → no higher than N")
4. Visually highlight the key criteria

Output the improved rubric.

What to fill in:

• {initial_rubric}: the barest version, "Score from 1 to 5 on criteria: X, Y, Z", or a ready detailed rubric
• {example_N}: the actual work to score (pitches, texts, code)
• {reference_N}: your correct score
• {error_analysis_N}: a short note on what the model missed or overrated

🚀 Quick start: paste into the chat:

Here's a Reflect-and-Revise template for improving scoring criteria. 
Adapt it to my task: [describe what needs to be scored]. 
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask which criteria matter for your task and what scale to use; it needs that to build the starting rubric. Then it will ask for 5-10 examples with your scores, the raw material for the first refinement iteration. The model will take the structure from the template and adapt it to your context.

⚠️ You need examples with reference scores: Full refinement takes 50-200 examples with correct answers. It also works with 5-10 examples, but the rubric will be less detailed and blind spots may remain.

⚠️ Not for subjective criteria: If the "correct score" depends on taste and opinion (creativity, humor, aesthetics), refinement can lock onto your personal preferences instead of objective patterns.

⚠️ Narrow optimization metric: The study only optimized scoring agreement (QWK). Other aspects of quality (soundness of rationales, fairness, consistency within categories) may not improve or may even get worse.

⚠️ Diminishing returns after 5-7 iterations: On ASAP, improvement continued until iteration 7-8, then plateaued. Further refinement gives minimal gains; the rubric hits the precision ceiling for that model.

The team used two essay datasets: TOEFL11 (12,100 essays, 1-5 scale) and ASAP (179 test essays, 1-6 scale). For each dataset they selected 200 examples: 100 for training (generating rubric improvements), 100 for validation (picking the best version), and the rest for testing.

They tested five models: GPT-4.1, GPT-5-mini, Gemini-2.5-Flash, Gemini-2.5-Pro, Qwen3-Next-80B. Each ran 10 iterations with batches of 10 examples. In each iteration the model scored essays, saw the gaps versus human scores and rewrote the rubric. The version with the best QWK on validation was selected.

They compared three starting points:

1. simplest_rubric: "Score from 1 to 6", a minimal instruction
2. simplified_human_rubric: a simplified version of the official rubric
3. human_rubric: the detailed official rubric for graders

The surprise: After refinement even the barest rubric beat the detailed human one on ASAP (+0.48 vs +0.26 QWK). That's counterintuitive; you'd expect a detailed rubric to be the best starting point. But it turns out LLMs are better at deriving criteria from their mistakes than at following ready-made human instructions.

The reasoning: Human rubrics are optimized for people: they contain context, examples, nuances of wording. LLMs often ignore the details or interpret them their own way. A simple rubric is a blank slate that the model builds out for itself by reflecting on real mistakes. The result fits the logic of that specific model more precisely.

On TOEFL11 the gain was smaller (+0.19 vs +0.47 on ASAP). The likely reason: TOEFL uses a coarse scale (3 levels: low/medium/high). With few gradations there's less room for improvement and it's harder to detect subtle error patterns. On ASAP's 6-point scale the rubric can draw finer distinctions.

Context: The researchers used this prompt for rubric refinement in their algorithm. The model received the current rubric and error data for a batch of examples, and generated an improved version.

You are an expert in designing scoring rubrics for automated essay scoring. 
Your task is to refine the current rubric based on the model's performance 
on a set of sample essays.

Current Rubric:
{current_rubric}

Performance Data:
{feedback_data}

The feedback_data contains:
- Essay text
- Model's predicted score and rationale
- True score assigned by human raters
- Observed discrepancy

Analyze patterns in the discrepancies and generate an improved rubric that:
1. Addresses systematic scoring errors
2. Adds concrete indicators for each score level
3. Specifies threshold values where applicable
4. Includes explicit conditional rules (e.g., "if X is absent, max score is N")

Output only the refined rubric text.

The feedback_data format:

[
  {
    "essay": "[essay text]",
    "predicted_score": 4,
    "rationale": "The essay demonstrates clear organization...",
    "true_score": 3,
    "discrepancy": "Overestimated organization quality"
  },
  ...
]
```

---

## Adaptations and Extrapolations

### 💡 Adaptation for other kinds of scoring

The method isn't just for essays; the principle applies anywhere, you need **repeatable scoring**:

**Scoring junior developers' code:**
```
Criteria: readability, efficiency, error handling
Examples: 10 PRs from juniors with your comments
Iterations: 3-5 (usually enough)

Result: A rubric with concrete rules
- "If there's no None handling → max 3/5"
- "If nesting is >3 levels deep → minus 1 point"
```

**Scoring social media posts:**
```
Criteria: engagement, readability, CTA
Examples: 20 posts with metrics (reach, clicks)
Iterations: 5-7

Result: A predictive engagement model
- "If there's a question in the first paragraph → +1 point"
- "If the CTA comes at the end with no context → -2 points"
```

---

### 🔧 Technique: Visualize the evolution → a transparent process

After each iteration, ask the model to show a **diff of the criteria**:
```
Show what changed in the rubric:

BEFORE:
- Criterion: Clarity of the problem (score 1-5)

AFTER:
- Criterion: Clarity of the problem (score 1-5)
  * 5 points: a concrete loss figure + target audience
  * 4 points: a clear statement without numbers
  * ...

WHY:
The model often overrated vague statements. Added a requirement for 
concrete numbers for the top scores.
```

This helps you **trace the logic** of the improvements and see which errors were critical.

---

### 🔧 Technique: Refine criteria one at a time → finer tuning

Instead of improving the whole rubric at once, improve **each criterion separately**:
```
Iteration 1: Improve only the criterion "Problem"
Examples: 10 cases where the model misjudged the problem

Iteration 2: Improve only the criterion "Solution"  
Examples: 10 cases where the model misjudged the solution

[...]

Iteration N: Combine all criteria into the final rubric

Pro: more targeted improvement; the model focuses on one aspect.

Con: slower (N iterations instead of 1), and it misses the interactions between criteria.

• Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revise (2025)
• Authors: Keno Harada, Lui Yoshida, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo
• The University of Tokyo
• Datasets: TOEFL11, ASAP (Automated Student Assessment Prize)
• Metric: Quadratic Weighted Kappa (QWK), a measure of scoring agreement