LLM-ReSum: A Summary the Model Grades and Improves Itself Until It's Good Enough

Everything happens in one chat, three requests in a row:

STEP 1 — Draft summary
  Request: "Summarize the document {document}"
  → An initial summary text

STEP 2 — Self-assessment
  Request: "Grade this summary on 4 criteria (1-5)
           + explain each score"
  → Scores for each criterion + concrete explanations

STEP 3 — Revision (if any criterion is < 4)
  Request: "Rewrite the summary, fixing the problems
           in the criteria scored below 4"
  → Improved text

Repeat steps 2-3 up to 3 times OR until all criteria score ≥ 4
If it doesn't converge, return the version with the best minimum scores

Task: Ethan is head of product at a fintech startup. An investor sent over a 40-page due diligence report on a competitor. He needs to get the gist across to the team in a 10-minute meeting: a dense summary with nothing important lost.

Prompt (3 sequential requests):

── REQUEST 1 ──
Summarize the following document.
Cover the key facts, conclusions and recommendations.

[report text]

── REQUEST 2 ──
Grade your summary above on four criteria.
For each, give a number from 1 to 5 and an explanation:

1. ACCURACY: are all facts from the original reflected
   correctly? any distortions or assumptions?

2. COMPLETENESS: are all the key ideas, data
   and conclusions from the document covered?

3. CLARITY: is it clear to someone
   who hasn't read the original?

4. OVERALL QUALITY: how well does the summary
   convey the essence of the document as a whole?

Format: [criterion] — [score/5] — [explanation]

── REQUEST 3 ──
Here is the original document and the current summary with scores.

Rewrite the summary, fixing the specific problems
in the criteria scored below 4.
Don't rewrite what's already good.
Focus only on the weak spots.

Result:

The model will show a draft, then an honest self-assessment like "Completeness — 3/5 — skipped the market-share section and didn't mention the key regulatory risk." Then in the third request it will fill in exactly the missing parts. The final summary will be denser and less "hand-wavy" than the first draft.

By default, LLMs are bad at self-criticism. When you give a task, the model optimizes for "give a good answer," not "give an honest assessment of my answer." Self-criticism isn't built into an ordinary request.

But models are good at grading against specific criteria. That's a different task: not generating, but checking. When you explicitly set the dimensions (accuracy, completeness, clarity) and ask it to justify each score, the model finds real problems instead of inventing praise.

The loop works because the grade becomes an instruction. Instead of an abstract "improve the text," you get a concrete "here's exactly what's wrong and why." That sharply narrows the room for edits. The model doesn't rewrite the good parts or guess what to improve; it fixes documented problems.

Control levers: - Quality threshold (≥ 4 out of 5) — lower it to 3 if draft quality is good enough. Saves requests - Maximum of 3 iterations — this guards against "semantic drift." After 3 edits the text starts losing touch with the original - 4 criteria — you can cut down to the 2-3 that matter most for your task (for example, just accuracy and completeness) - Grading output format — ask it to add concrete examples of problems with quotes from the text → revisions get more precise

── STEP 1: DRAFT ──

Summarize the following {document_type}.
Cover the key facts, conclusions and {what_matters_for_the_task}.

{text}

── STEP 2: SELF-ASSESSMENT ──

Grade your summary on {number_of_criteria} criteria (1-5).
For each, give a score and a concrete explanation:

{criterion_1}: {criterion_definition}
{criterion_2}: {criterion_definition}
{criterion_3}: {criterion_definition}
{criterion_4}: {criterion_definition}

Format: [criterion] — [score/5] — [what exactly is wrong]

── STEP 3: REVISION ──

Here is the source document and the summary with scores.
Rewrite the summary, fixing the problems in the criteria
scored below {threshold}.
Don't touch anything that scored {threshold} or higher.

What to fill in: - {document_type} — "contract," "research paper," "investment memo" - {what_matters_for_the_task} — "risks and terms," "methodology and conclusions," "key metrics" - {criterion_N} — the four from the study (accuracy, completeness, clarity, overall quality) or your own - {criterion_definition} — a short explanation of what to check - {threshold} — 4 is recommended; 3 is fine for quick tasks

🚀 Quick start — paste into the chat:

Here is the LLM-ReSum template: iterative improvement
of a summary through self-assessment. Adapt it to my task:
{your task}. Ask questions to fill in the fields.

[paste the template above]

The LLM will ask which document needs summarizing and which criteria matter, because without that it doesn't know what to base the grading on.

⚠️ Long documents: With very long texts (20K+ words) the model can lose the thread by the third iteration. Split the document into meaningful chunks and run the method on each part.

⚠️ Subjective criteria: The method works better for verifiable criteria (factual accuracy, coverage) than for matters of taste (style, tone of voice). "Clarity" is already borderline.

⚠️ Hallucinations during iterations: Every revision carries a risk of "filling in the blanks." The 3-iteration cap isn't arbitrary; it's a safeguard. Don't lift it.

⚠️ Self-assessment ≠ objective assessment: The model may inflate its scores. If the result seems too easy, add "be critical; don't give 4-5 without real grounds" to the prompt.

A team from the University of North Texas started with a diagnosis: they took 14 popular evaluation metrics (ROUGE, BLEU, BERTScore and others) and checked how well they agree with real people's judgments. To do that they assembled 7 datasets from 5 different domains, from short news items to 27,000-word research papers and government documents, with 1500+ human ratings.

The verdict was blunt: ROUGE and BLEU correlate with humans weakly or even negatively. In other words, a high ROUGE score doesn't mean a person will find the summary good. This is especially noticeable for LLM summaries: people prefer fresh phrasing, and ROUGE punishes it for "differing from the reference."

Then they tested LLMs as graders, and models as judges work significantly better than traditional metrics, especially for language quality. An interesting effect: multi-agent grading (several different LLMs grade independently, then the scores are aggregated) is more stable than a single agent, the same mechanism as with several human experts.

The final test of LLM-ReSum ran across three domains. The iterative loop improved accuracy by up to 33% and completeness by up to 39%. In blind comparisons, people preferred the processed summaries in 89% of cases. Importantly, all of this happened without fine-tuning the model: just prompts and loops.

The researchers described the LLM-ReSum algorithm as pseudocode. Here's the exact logic:

Input: Document D, functions f_sum, f_eval, f_refine
       Quality threshold τ, Max iterations T_max
Output: Enhanced summary S*

S[0] ← f_sum(D)          // Generate a draft summary
t ← 0

while t < T_max do:
  q[t], r[t] ← f_eval(D, S[t])   // Evaluate: scores + rationales

  if min(q[t]) ≥ τ then
    return S[t]           // All criteria above threshold: done

  I ← {i | q[t][i] < τ}  // Find the weak dimensions
  F[t] ← feedback({q[t][i], r[t][i]} for i in I)  // Build the feedback
  S[t+1] ← f_refine(D, F[t])  // Refine based on the feedback
  t ← t + 1

return argmax_t' min_i(q[t'][i])  // Return the best version

Context: The researchers ran this through an API, but the logic itself (three kinds of prompts: generation, grading, revision, plus a threshold-based exit condition) can be reproduced by hand in an ordinary chat.

💡 Adaptation for checking any text (not just summaries):

The same loop works for grading and refining a sales proposal, an article, a letter to an investor, a technical spec. Only the criteria change to fit the task.

── GRADING ──

Grade the following {text_type} on these criteria (1-5):

PERSUASIVENESS: how strongly does the text prompt action?
SPECIFICITY: are there facts, numbers, examples?
NO FLUFF: are there unnecessary words that carry no meaning?

For each, give a score and what exactly is wrong.

── REVISION ──

Rewrite the weak spots (score below 4)
without touching what already works.

🔧 Technique: add a step that visualizes the scores

After the self-assessment, ask the model to output the scores as a table. That way the pattern jumps out: which criterion consistently sags across different documents.

After grading, output a table:
| Criterion | Iteration 1 | Iteration 2 | Iteration 3 |
|-----------|-------------|-------------|-------------|
| ...       |             |             |             |

If "Completeness" is always 2-3, your generation prompt compresses too hard. Time to rewrite step one.

🔧 Technique: name the reason for every edit explicitly

In the revision step, add the instruction "for each change, write in parentheses which problem from the grading it fixes." That makes the edits traceable: you can see the link between diagnosis and treatment.

Paper: LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluation (2024)

Authors: Huyen Nguyen (Cigna Group / Evernorth Health Services), Haoxuan Zhang, Yang Zhang, Haihua Chen, Junhua Ding — University of North Texas

Benchmark: PatentSumEval, an annotated dataset for evaluating summaries of patent documents

Evaluation platform: APPEN (https://client.appen.com)

Metrics mentioned: ROUGE, BLEU, BERTScore, MoverScore, SummaC, QuestEval