MedRGAG: Three-Stage Knowledge Gap-Filling via Summarization, Gap Analysis, and Context Generation

STEP 1: Summarization
For each retrieved document → extract what's useful for the question
(filter noise, drop the irrelevant)

STEP 2: Gap analysis
Read all summaries → identify 2-3 key knowledge gaps
(what's CRITICAL for the answer but missing)

STEP 3: Context generation
For each gap → generate a reference document
(fill in what's missing)

STEP 4: Document selection
Group retrieved + generated by topic → pick the top 5 sources
(balance: facts from retrieval + context from generation)

STEP 5: Final answer
Use the 5 selected sources → give the answer

Each step runs as a separate request. Every stage is a new prompt that takes the previous one's results.

Task: You're a consultant helping a startup choose a business model for a B2B SaaS product. You have a few articles on monetization, but the information is scattered.

Prompt:

I have 3 articles on SaaS monetization:

[Article 1]: The freemium model drives high conversion in paid segments...
[Article 2]: Enterprise sales require a long deal cycle...  
[Article 3]: A PLG approach lowers CAC through virality...

TASK: Determine the optimal monetization strategy for a B2B reporting-automation tool (target customers: mid-sized businesses in the US, 50-500 employees).

Do the following in order:

1. SUMMARIZATION: For each article, pull out only what's relevant to my case (mid-market B2B, US). If an article has nothing useful, write "No useful information".

2. GAP ANALYSIS: Read all the summaries. Identify 2-3 critical knowledge gaps — what do I need to know to choose a model that's missing from the articles?

3. CONTEXT GENERATION: For each gap, write a short brief (3-4 sentences) with the missing information.

4. SELECTION: From all sources (3 articles + briefs), pick the top 5 most relevant for the decision. Explain why these.

5. FINAL ANSWER: Based on the selected sources, recommend a monetization strategy.

Result:

The model will produce a structured process:

• Step 1: A summary of each article focused on mid-market B2B
• Step 2: For example, "Gap 1: no data on mid-sized US businesses' willingness to pay", "Gap 2: the risks of freemium in B2B aren't covered"
• Step 3: Short briefs for each gap
• Step 4: A list of the chosen sources with the rationale
• Step 5: A final recommendation grounded in the sources

You get explicit reasoning behind the decision — you can see where each idea came from (an article or a generated brief) and why the model reached its conclusion.

The LLM's weakness: The model doesn't know what it doesn't know. Asked directly, it either makes things up (hallucinates) or ignores the knowledge gaps and gives a shallow answer.

The LLM's strength: The model is great at analyzing what's there and logically inferring what's missing — if you explicitly ask it to. It's also good at generating context on a specific request ("write a brief on X"), but bad at generating "in general".

How the method uses that strength:

1. Summarization filters out noise — instead of "here are 10 documents, sort it out", the model first extracts the essence of each one. That lowers the cognitive load at the final stage.
2. Explicitly identifying gaps turns an abstract task ("give a good answer") into a concrete one ("what exactly is missing?"). The model stops making things up and honestly says "I don't know about X".
3. Targeted generation creates context pinpoint-style ("a brief on X") rather than broadly ("tell me everything about the topic"). This sharply reduces hallucinations, because the request is narrow and checkable.
4. Selection keeps the model from drowning in information — it picks the top 5 sources itself, balancing facts (from retrieval) and logic (from generation). Without this step the model may overrate the generated context (it reads more coherently) or underrate the retrieved facts (they're fragmentary).

Prompt control levers:

• Number of gaps (2-3) → raise to 5 for complex tasks, drop to 1 for simple ones — controls the depth of analysis
• Number of final sources (top 5) → more sources = more context, but a higher risk of noise
• Generation format ("3-4 sentences") → set the brief length for your task
• Selection criteria → add "prioritize facts from retrieval" or "prioritize recent data"

I have {information_sources}.

TASK: {your_task}.

Do the following in order:

1. SUMMARIZATION: For each source, pull out only what's relevant to the task. If a source has nothing useful, write "No useful information".

2. GAP ANALYSIS: Read all the summaries. Identify 2-3 critical knowledge gaps — what do I need to know to solve this that's missing from the sources?

3. CONTEXT GENERATION: For each gap, write a short brief (3-4 sentences) with the missing information.

4. SELECTION: From all materials (original sources + briefs), pick the top 5 most relevant for the solution. Explain why these.

5. FINAL ANSWER: Based on the selected sources, solve the task.

Placeholders:

• {information_sources} — documents, articles, notes, or "I ran a search and found..."
• {your_task} — a specific question or problem

If you DON'T have sources ready:

Replace Step 1 with:

1. SEARCH + SUMMARIZATION: Find relevant information on the topic in your own knowledge. Split it into 3-4 thematic blocks and summarize each.

The model uses its internal knowledge as "pseudo-documents" and follows the same scheme from there.

⚠️ Number of requests: The method takes 1 request for all steps (they fit in one prompt) or 5 separate requests (if you want to control each stage). For simple questions this is overkill — a direct answer is faster.

⚠️ Generation quality: In Step 3 the model may generate inaccurate briefs if the gap is beyond its knowledge. In medicine that's critical (wrong fact → wrong diagnosis); in business it's tolerable.

⚠️ Overhead on simple tasks: If the question is simple ("capital of France?"), the 5-step process hurts — the model may "find" gaps that don't exist and overcomplicate the answer.

⚠️ Subjective criteria: The method works worse when you need judgment (creativity, style) rather than facts. The model doesn't know which gap is more critical: "missing data on metric X" vs "missing expert Y's opinion".

The team took five medical datasets (medical licensing exams, PubMed questions) — about 8,000 questions in total. They compared three approaches: RAG (retrieval from medical textbooks + Wikipedia), GAG (the model generating context "from memory"), and MedRGAG (their method).

They tested on three reader models (Qwen2.5-7B, LLaMA-3.1-8B, Ministral-8B) — the models that read the context and give the final answer. Generation used LLaMA-3.1-8B; the helper steps (summarization, gap analysis, selection) used GPT-4o-mini.

Why the results look like this:

In medicine, documents are often fragmentary — one covers symptoms, another treatment, a third the disease mechanism. RAG returns all three and the model gets lost in the volume. GAG generates coherent text but invents details (such as a wrong dosage).

MedRGAG first reads what was found, realizes "symptoms are covered, but the disease mechanism isn't", generates specifically the disease mechanism, and then chooses: it takes the factual symptoms from retrieval + the generated mechanism. That gives +12.5% accuracy over pure RAG.

The most interesting part: when they removed document selection (Step 4) and just mixed all 10 sources (5 retrieved + 5 generated), accuracy dropped. It turned out the reader model overrates the generated documents (they read more coherently) and ignores the retrieved facts (they look scattered). Selection balances this — it explicitly says "take 2 from retrieval, 3 from generation".

Why the method works better:

Classic RAG takes the top 5 documents by similarity to the question. But similar ≠ useful. You might get 5 documents on symptoms and none on treatment. MedRGAG analyzes what was found and deliberately generates what's missing.

Classic GAG generates 5 documents "about the question in general". The model invents details because it doesn't know what exactly is needed. MedRGAG first identifies the gaps ("I need information about X"), and only then generates — which reduces hallucinations.

1. Simplified single-request version

If you don't want to make 5 separate requests, combine everything into one prompt with an explicit structure:

Task: {your_task}

Source materials: {sources}

Do this as an internal monologue (don't show me the intermediate steps):
1. Summarize each source
2. Identify 2-3 knowledge gaps  
3. Generate briefs for the gaps
4. Pick the top 5 sources (original + briefs)

In the final answer show:
- Which sources you used (mark retrieved vs generated)
- The answer to the task

The model will run all the steps internally but show only the result. The downside: you lose control over each stage.

2. Simplified two-stage version

For situations where you have no sources at all (working purely from the model's knowledge):

STEP 1:
Topic: {your_topic}

Find all relevant information in your knowledge. Split it into 3-4 thematic blocks. For each block, state WHAT you know and what's MISSING for the full picture.

STEP 2:  
Based on the gaps from Step 1, write the missing briefs. Use the blocks + briefs to answer the question: {your_question}.

This is an adaptation for when there are no external documents but you still need structured thinking.

3. Combining with Chain-of-Thought

If the task needs not just knowledge but also logical steps (say, diagnosing a problem):

Task: {your_task}

Materials: {sources}

1. KNOWLEDGE: Run MedRGAG (summarization → gaps → generation → selection)

2. REASONING: Based on the selected sources, reason step by step:
   - What is known?
   - What are the hypotheses?
   - How can we test them?
   - Final conclusion

Show both processes separately.

This lets you separate knowledge gathering (MedRGAG) from logical inference (CoT).