MODE 1: Self-refinement (doesn't work)
Request → Answer → "Improve it" → [model guesses what to fix] → ~30% quality

MODE 2: Guided refinement (works)
Request → Answer → "Fix: item A, item B" → [model knows what to repair] → 90%+ quality

MODE 3: Partially guided (in between)
Request → Answer → "Fix: item A (find the rest yourself)" → [partial success]

The key lever: specificity of feedback. The more precisely you point out the problem, the better the result.

Task: You've written a sales proposal for a B2B client and got a first draft from Claude. You want to make it better.

❌ Ineffective prompt (self-refinement):

Improve this sales proposal. Make it more persuasive.

Result: The model will make cosmetic edits and maybe add some fluff, but it won't touch the real problems. Quality goes up by 1-3%.

✅ Effective prompt (guided refinement):

Revise the sales proposal on these points:

1. The "Problem" section has no concrete figures for the client's losses — add a calculation
2. The "Why us" block sounds like everyone else's — highlight 1 unique advantage
3. The call to action is vague ("get in touch") — propose a specific next step with a deadline
4. There's no social proof — mention a case study or testimonial

Leave everything else as is.

Result: The model will work through each item in turn. You'll see structured changes that follow the list. Quality lands at 85-95% of ideal.

The LLM's weakness: Models are bad at diagnosing their own errors. When you ask them to "improve", the model has to: (1) evaluate the current answer, (2) find the weak spots, (3) fix them. Steps 1-2 are where it falls apart. The study showed that even reasoning models (DeepSeek-R1, o1), after their first answer, stop looking for new problems: they "check the logical consistency" of what they've already written instead of searching for what they missed.

The LLM's strength: Models are excellent at executing specific instructions. If you say "fix item X", the model knows what to do and succeeds 95%+ of the time. That's instruction-following, a skill that's highly developed in modern models.

How the method uses this: You move diagnosis outside the model: you take on the role of "critic" and leave the model the role of "executor". The model doesn't guess what to fix; it gets a clear list and works through it.

Control knobs:

• Number of items: 3-5 specific notes work better than 1 general one or 15 tiny ones
• Wording: "Add X" works better than "X is missing" (a positive instruction)
• Constraint: "Leave everything else as is" protects against unnecessary changes
• Iterations: 3-5 rounds of feedback get you to the quality plateau

Revise the {content_type} on these points:

1. {specific problem 1} — {what to do}
2. {specific problem 2} — {what to do}
3. {specific problem 3} — {what to do}

Don't change anything else.

Placeholders:

• {content_type} — what you're revising: an article, email, code, a presentation
• {specific problem} — what's wrong: "no numbers", "too long", "confusing structure"
• {what to do} — the action: "add", "cut", "break into sub-points", "replace with..."

Checklist prompt for diagnosis (if you don't know what to fix yourself)

Here is the {content_type}:

{text}

Evaluate it against this checklist (answer Yes/No for each item):

1. {criterion 1}?
2. {criterion 2}?
3. {criterion 3}?
4. {criterion 4}?
5. {criterion 5}?

For every "No", write what specifically needs fixing.

This prompt turns diagnosis into an explicit step. You get a list of problems, then use it for guided refinement.

⚠️ Domain matters: On legal tasks, self-refinement works better (+5-8% for top models). In math it barely works at all. If your task involves law, contracts, or regulation, self-refinement is worth a shot.

⚠️ "Think longer" doesn't help: Increasing the thinking budget doesn't improve self-refinement. The model just writes more text without finding new problems.

⚠️ Reasoning models don't save you: DeepSeek-R1, o1, and Claude with extended thinking show the same problems. They reflect heavily in the first answer, but when asked to "improve", they just check the logic of what's written instead of looking for new errors.

⚠️ Partial feedback = partial results: If you point out 3 problems out of 6, the model will fix those but will most likely miss the remaining 3.

A team from KAIST, CMU, and NVIDIA collected 1,000 tasks from real-world sources: Korean university essays, California Bar exams (law), Stanford problems (math/statistics), and Humanity's Last Exam. Each task comes with a checklist of 9-10 items, the criteria for a quality answer.

They tested 34 models in two modes: self-refinement (5 iterations of "improve it") and guided refinement (at each iteration the model gets the list of unmet checklist items). Grading used GPT-4.1 as a judge working from the checklist.

The surprise: The gap between the modes was enormous. Claude-Opus-4.1 in guided mode reaches 98.4% by round 5; in self-refinement it gets stuck at 19.5%. And when models were given only the checklist (what to check, but not how to fix it), quality doubled: from 31% to 76% for Gemini 2.5 Pro. That proves the problem is specifically diagnosis, not the ability to fix things.

DeepSeek-R1 showed an interesting pattern: lots of self-correction in the first answer ("Wait", "Alternatively"), with 69.7% of reasoning tokens spent in the first round. In later rounds the model just "checks logical consistency" without searching for new problems. Hence the negative trend: -0.1% over 5 iterations.

💡 Adaptation: A two-step revision loop

If you don't know what to fix, split it into two steps:

Step 1 — Diagnosis:

Here is my text: {text}

Imagine you're an editor. Find the 3-5 biggest problems in this text. 
For each problem:
- What's wrong
- Where exactly in the text
- How to fix it (one specific action)

Step 2 — Fixing (a new prompt or a follow-up):

Now fix the text based on these points. Leave everything else as is.

You move diagnosis into a separate step where the model acts as an outside critic rather than grading "its own" work. That gets around its blindness to its own errors.

🔧 Technique: Checklist-driven iterations

For systematic revision, build a reusable checklist for each content type:

# Checklist for a sales proposal

1. Is there a specific client problem with loss figures?
2. Is there 1 unique advantage (not "quality and experience")?
3. Is there a specific case study/testimonial?
4. Is there a clear next step with a deadline?
5. Does it fit on 1 page?

Run this checklist after every draft → get your list of "No"s → turn it into guided refinement.

Paper: "RefineBench: Evaluating Refinement Capability of Language Models via Checklists"

Authors: Young-Jun Lee, Seungone Kim, Byung-Kwan Lee, Minkyeong Moon, Yechan Hwang, Jong Myoung Kim, Graham Neubig, Sean Welleck, Ho-Jin Choi

Institutions: KAIST, Carnegie Mellon University, NVIDIA

Links:

• Website: https://passing2961.github.io/refinebench-page/
• Code: https://github.com/RefineBench/refinebench-eval
• Dataset: https://huggingface.co/datasets/RefineBench/RefineBench