STEP 1: Initial summary           → a short overview of the document
         ↓
STEP 2: Question generation       → 5-10 questions about the ORIGINAL (not the summary)
         ↓
STEP 3: Check                     → can they be answered from the summary?
         ↓
STEP 4: Fill in                   → add info to the summary for unanswered questions
         ↓
      [Repeat steps 2-4: 5-10 times]
         ↓
FINAL: An information-dense summary

Each step is a separate LLM request. A full cycle: 20-40 requests (but the tokens are short).

Task: Prepare a lease agreement (15 pages) for quick consultations with Claude, so you can later ask questions and get accurate answers without rereading the whole document.

Prompt (Step 1, initial summary):

Here is a lease agreement. Create an information-dense summary: all key terms, deadlines, amounts and obligations of the parties. Goal: this summary should be enough to answer any question about the agreement.

[agreement text]

Prompt (Step 2, question generation):

Here is the original agreement. Generate 5 diverse questions that it's important to be able to answer from this document. Format:
Q: [question]
A: [short answer, a word or phrase]

[agreement text]

Prompt (Step 3, check):

Here is a summary of the agreement. Try to answer the questions based ONLY on the summary. If there's no answer, write "not in summary".

Summary: [current summary]

Questions:
[list of questions from step 2]

Prompt (Step 4, fill in):

Improve the summary so it contains the information needed to answer these questions. Keep everything important from the current version.

Original agreement: [text]
Current summary: [summary]
Unanswered questions: [unanswered questions]

Result: After 3-5 cycles you'll have a 200-400 word summary covering: rent, term, termination conditions, penalties, repair obligations and renewal procedure. You can ask questions against this summary and get accurate answers without loading the whole agreement into every request.

The LLM weakness: the "lost in the middle" curse: models make poor use of information in the middle of a long context. At 11,000 tokens, facts in the middle of a document are often ignored. On top of that, a standard summary is optimized for "brevity", not for "usefulness for answering questions".

The LLM strength: Models are great at generating questions about a text and checking whether a passage contains the answer. That's a much clearer task than an abstract "pull out the main points".

How the method connects them: Instead of a subjective "what matters", there's an objective criterion: can you answer the question? Each iteration finds a gap (an unanswered question) and closes it. The summary evolves under the pressure of concrete information needs.

Control levers:

• Number of iterations (5-10): more = denser, but diminishing returns after 10
• Questions per iteration (1-2 is better than 10): gradual improvement beats bulk edits
• Question type: you can steer it with "questions about deadlines", "questions about money" to focus on the aspects you need

Step 1: Initial Summary

Create an information-dense summary of the document. Goal: the summary should be enough to answer questions without going back to the original.

Document:
{document_text}

Step 2: Question Generation

Generate {number_of_questions} diverse questions about the document. The questions should cover important information. Answers should be short (a word or phrase).

Format:
Q: [question]
A: [short answer]

Document:
{document_text}

Step 3: Check

Answer the questions based ONLY on the summary. If there's no answer, write "not in summary".

Summary:
{current_summary}

Questions:
{question_list}

Step 4: Fill In

Improve the summary by adding the information needed to answer the unanswered questions. Keep what's important from the current version. Keep the summary short.

Original: {document_text}
Current summary: {summary}
Unanswered questions: {unanswered_questions}

Return: Updated summary:

Placeholders:

• {document_text}: the full text to summarize
• {number_of_questions}: 3-5 per iteration
• {current_summary}: the result of the previous step
• {question_list}: the Q&A from step 2
• {unanswered_questions}: the ones the summary couldn't answer

🚀 Quick start: paste this into the chat:

I want to create an information-dense summary of a document using the Chain of Summaries method.

Method: iteratively improve the summary through the loop "questions → check → fill in".

Here is the document: [paste document]

Walk me through 5 iterations:
1. Create an initial summary
2. Generate 3 questions about the original
3. Check whether the summary answers them
4. Fill in what's missing
5. Repeat steps 2-4 four more times

After each iteration, show the current summary and which gaps were closed.

The LLM will take the document and run the full cycle, showing how the summary evolves from the first version to the final one.

⚠️ Lots of requests: A full cycle is 20-40 LLM calls. For a one-off task it's easier to write one good summary. CoS pays off when the summary will be reused many times.

⚠️ Fact-heavy content: The method is optimized for Q&A, so it works well for contracts, manuals and technical documentation. It's weaker for narratives and fiction, where the "main point" isn't the facts.

⚠️ You need the original: Each iteration requires access to the full document to generate questions and check them. You can't improve a summary "blind".

⚠️ Weaker on TruthfulQA: On a dataset with nuanced claims (not pure facts) the method shows a smaller gain: ROUGE-L 14.77 vs. 8.89, but that's still +66%.

The team tested CoS on three Q&A datasets: TriviaQA (factual questions), TruthfulQA (truthfulness), SQuAD (extracting answers from context). They tested three models, from commercial to local: GPT-4o-mini, Llama-3.2:3B, Qwen-2.5:7B.

The main question: can you answer questions from a summary better than from the full document? It turns out you can. On TriviaQA with GPT-4o-mini, the CoS summary scored 0.80 F1 versus 0.76 for the full document. And the summary takes up 170 tokens vs. 11,000 for the original. That's counterintuitive: less information → better results. The reason is that the "lost in the middle" curse goes away.

An interesting finding about hyperparameters: 10 iterations with 1 question each consistently beat 1 iteration with 50 questions (0.64 vs. 0.49 F1 for Llama). Gradual improvement works better than a bulk rewrite. They also compared synthetic and human-written questions: almost no difference (+0.07 vs. +0.05), which means the method works without ready-made Q&A pairs.

An attempt to compress the multi-step process into one model (fine-tuning a small Qwen with GRPO) did worse: 0.74 vs. 0.80. Iteration is essential and can't be replaced by training.

Question-generation prompt:

Generate {number_of_questions} diverse and specific questions in Q: format based on the content. Do not include question numbers. Each question should target important information from the text. Each answer should be concise (word or short phrase) and directly address the question.

Ensure answers are brief (a word or short phrase, not a full sentence) and factually accurate based on the text. Format should be:
Q: <question>
A: <answer>

Summary-refinement prompt:

You are an expert text summarizer. Your task is to refine an existing summary to address specific user questions.
Rules:
- Include information that directly answers the user's questions
- Preserve relevant key points from the original summary
- Return the original summary unchanged if it already contains all necessary information
- Return the original summary if the questions are not relevant to the text
- Keep the summary short and concise
- Don't include questions in the summary.
- Every time start with: Updated Summary:

Knowledge Base Passage: {passage}
Existing Summary: {existing_summary}
Questions to Address: {antithesis_questions}
Provide an updated summary addressing the questions while maintaining the informational content of the original summary.

Context: These prompts were used to automate the loop in the experiments. When doing it by hand, you can simplify the wording.

💡 Adaptation: CoS for building a knowledge base

If you work on a project with an LLM and keep uploading the same documents (policies, specs, research), run them through CoS once. You'll get compact "cards" that take up less context and produce better answers.

I have 5 documents I use often when working with you:
1. [title] — [short description]
2. ...

For each one, create a CoS summary (3 iterations). Save the result as "Project X Knowledge Base". In future conversations I'll paste this knowledge base instead of the full documents.

🔧 Technique: targeted questions for focus

Instead of "generate any questions", steer it toward the aspect you need:

Generate 5 questions about the document, focusing on:
- Financial terms and penalties
- Timelines and deadlines
- Liability of the parties

This gives you a summary optimized for your typical queries.

🔧 Technique: mini-CoS in 3 steps

For quick use without 10 iterations:

1. Create a summary of the document
2. Come up with 5 tricky questions the summary might NOT answer
3. Check and fill in the summary in one pass

[document]

It doesn't give maximum quality, but it catches the main gaps in a single request.

Paper: "Chain of Summaries: Summarization Through Iterative Questioning"

Authors: William Brach (Slovak Technical University), Lukas Galke (University of Southern Denmark)

Datasets: TriviaQA, TruthfulQA, SQuAD v1.1

Related methods: SELF-REFINE (iterative improvement), SummIt (summarization + self-evaluation), Hegel's dialectical method