PRISM: A Three-Agent System for Precise Information Retrieval

STEP 1: Question Analyzer
Input: complex question
Output: list of sub-questions (each = a fact needed for the answer)

STEPS 2-4: Selector ⇔ Adder loop (2-3 iterations)
Each iteration:

  2.1 Selector
  Input: sub-questions + candidates (20-50 passages)
  Output: only 100% relevant ones → high precision, may lose something important

  2.2 Adder
  Input: sub-questions + candidates + what the Selector picked
  Output: missed but necessary evidence → high recall

Iterations are merged and deduplicated → final evidence set

STEP 5: Answer Generator
Input: question + final evidence set
Output: answer

Every step runs as a separate prompt. Lists of selected passages are passed between the Selector and the Adder.

⚠️ A method for hard problems: PRISM works best when the question requires gathering information from different sources and combining facts. It's not for simple questions like "capital of France" — one search query is plenty there.

Task: Help a client pick a new logistics provider in the US. Requirements: operates in the Midwest, has cold-storage warehouses, is an Amazon partner, launched after 2020, has good reviews.

Prompt for the Question Analyzer:

Break this complex question into a list of simple sub-questions.
Each sub-question should target one specific fact.

Question: Which US logistics companies operate in the Midwest,
have cold-storage warehouses, are Amazon partners, launched after 2020
and have good reviews?

Answer format:
1. [sub-question 1]
2. [sub-question 2]
...

Prompt for the Selector:

You are given a list of sub-questions and a list of candidate passages.
Select ONLY the passages that DIRECTLY answer at least one sub-question.
Be strict — if in doubt, leave it out.

Sub-questions:
1. Which logistics companies operate in the Midwest?
2. Which of them have cold-storage warehouses?
3. Which are Amazon partners?
4. Which launched after 2020?
5. Which have good reviews?

Candidates:
[1] "Heartland Freight" operates out of Chicago...
[2] "ColdRoute" specializes in refrigerated trucking...
[3] Amazon expands its partnership with "FulfillPlus"...
[20 passages]

Answer: only the numbers of the selected passages, comma-separated

Prompt for the Adder:

You are given a list of sub-questions, candidates and ALREADY SELECTED passages.
Check: are any sub-questions still unanswered?
Add passages that COMPLEMENT the selection.

Sub-questions: [the same 5]
Already selected: [1, 3, 7]
Candidates: [the same 20]

Add the numbers of NEW passages (don't duplicate the selection)

Result:

After 2-3 iterations of the Selector ⇔ Adder loop, the model returns 4-6 passages — each answers a specific sub-question, no duplicates, no junk. The Answer Generator gets a compact context and produces a precise list of companies with justification.

The LLM weakness: Models can't balance precision and recall at the same time. If the prompt says "find everything relevant," the model drags in 50 passages, half of them junk. If it says "only exact matches," it skips important but non-obvious connections. Trying to combine both in one prompt gets you a mediocre result: neither precision nor recall.

The LLM strength: Models follow clear roles extremely well. If the prompt says "you are a strict judge, discard anything doubtful," the model filters hard. If it says "you are the champion of completeness, look for gaps," the model finds what was missed. Models are also good at step-by-step reasoning — they can analyze what's already selected and what's still missing.

How PRISM leverages this: The method splits conflicting goals into separate prompts. The Selector gets the "maximum precision" role — it cuts without mercy. The Adder gets the "maximum recall" role — it hunts for holes. Iterations let the two gradually converge: in round one the Selector chops off the obvious junk and the Adder brings back the obviously missed. Round two is subtler — the Selector sees what the Adder added and can trim the excess, while the Adder checks whether any gaps remain. By round three the set stabilizes.

Prompt control levers:

• Number of iterations (N): The paper uses 3; you can drop to 1-2 for simpler tasks (fewer calls) or raise to 4-5 for especially hard ones
• Selector strictness instructions: Loosen them ("include if there's even a hint") for tasks where recall matters more, or tighten them ("direct matches only") for tasks where precision is critical
• Adder's role: Give it a specific focus like "look only for temporal links" or "look only for quantitative data" to fit the task
• Sub-question format: You can ask the Question Analyzer for a more granular decomposition (10 sub-questions) or a coarser one (3 sub-questions)

Question Analyzer

Break the complex question into a list of simple sub-questions.
Each sub-question = one specific fact needed for the answer.

Question: {your_question}

Answer format:
1. [sub-question targeting fact 1]
2. [sub-question targeting fact 2]
...

Selector

You are given a list of sub-questions and candidates (passages/documents).
Select ONLY the candidates that DIRECTLY answer at least one sub-question.

Rule: if in doubt, leave it out. Better to miss one than to add something irrelevant.

Sub-questions:
{sub_question_list}

Candidates:
{numbered_candidate_list}

Answer: only the numbers of the selected candidates, comma-separated

Adder

You are given a list of sub-questions, candidates and ALREADY SELECTED items.

Your task: check whether any sub-questions are still unanswered.
Add candidates that COMPLEMENT the current selection and close the gaps.

Rule: don't duplicate what's already selected. Add only what is truly needed.

Sub-questions:
{sub_question_list}

Already selected:
{selector_picked_numbers}

Candidates:
{all_candidates_list}

Answer: numbers of NEW candidates to add (or "none" if everything is covered)

Iteration Loop

Round 1:
→ Selector picks from all candidates
→ Adder reviews the Selector's picks, adds what was missed
→ Merge Selector picks + Adder additions = set_round_1

Round 2:
→ Selector reviews set_round_1, removes the excess
→ Adder reviews the Selector's result, adds anything missing
→ Merge = set_round_2

Round 3:
→ repeat the logic
→ Final set = deduplicated union of all rounds

Answer Generator

You are given a question and a set of evidence (passages/facts).
Answer the question using ONLY information from the evidence.

Question: {original_question}

Evidence:
{final_set_after_iterations}

Answer:

How to fill it in:

• {your_question} — a complex question that requires combining facts from different sources
• {sub_question_list} — the Question Analyzer's output
• {numbered_candidate_list} — numbered passages/documents from search results or a knowledge base
• {selector_picked_numbers} — the list of numbers the Selector returned

🚀 Quick start — paste into the chat:

Here is the PRISM template — a method for precise information retrieval with three agents.
Adapt it to my task: [describe your task — what question, from which sources].
Ask clarifying questions so you can fill in the agent templates correctly.

[paste the Question Analyzer, Selector and Adder templates above]

The LLM will ask about the format of your sources (passages? articles? documents?), how many candidates you typically get back, and how strict the filtering should be — because the Selector's instructions depend on that. It will take the three-agent pattern and adapt the prompts to your task.

⚠️ Not for simple questions: The method is overkill if the question can be answered with a single search query. The overhead of 5-7 LLM calls only pays off for multi-hop tasks.

⚠️ Depends on candidate quality: If the initial search returned 20 passages and the right one isn't among them, the Adder can't help. The method cleans up and fills in the set, but it doesn't go back to search the corpus again.

⚠️ Cost of iterations: 3 rounds of Selector ⇔ Adder = at least 6 calls + Question Analyzer + Answer Generator. For budget scenarios you can cut it to 1 round, but quality drops.

⚠️ Subtle logical chains: If the link between facts isn't obvious (e.g., "A was born in city B, B was founded by king C, C fought a war with D"), the model may skip intermediate links. The method shines on explicit multi-hop connections.

The team used four multi-hop QA benchmarks: HotpotQA, 2WikiMultiHopQA, MuSiQue, MultiHopRAG. They compared against strong baselines: IRCoT (iterative retrieval + chain of thought), SetR (set-wise reranking via an LLM), RankZephyr and RankGPT (top-tier rerankers). They used GPT-4o, Gemini-2.5-Flash and DeepSeek — confirming that the method works across different models.

The evaluation idea was simple: first measure retrieval quality (precision/recall at the passage and sentence level), then measure QA accuracy (Exact Match and F1). Three scenarios were compared: (1) the model answers from all candidates, junk included, (2) the model answers from PRISM's cleaned-up set, (3) the model answers from the gold evidence (upper bound).

Why the results came out the way they did:

On MuSiQue (the hardest benchmark, with 2-4 hops) PRISM delivered the biggest gain: recall of 83% vs 57% for IRCoT. The reason: MuSiQue questions are deliberately built so you can't take shortcuts. IRCoT drags in tons of noise trying to catch everything, while PRISM finds the balance through its Selector → Adder loop. On HotpotQA the gain is smaller (91% vs 73%), because the questions are easier and IRCoT already does a decent job.

The surprise: On 2WikiMultiHopQA IRCoT was slightly stronger at QA (57.7 EM vs 48.6 for PRISM), even though PRISM's retrieval was better. The analysis showed that the dataset has lots of compositional questions where IRCoT wins thanks to few-shot examples in its QA agent, while PRISM ran zero-shot. But on precision PRISM wins everywhere: 83-91% versus 60-70% for the baselines. That means PRISM produces compact sets — fewer tokens, less noise, fewer hallucinations.

Practical takeaway: If a task requires collecting 3-4 facts from different places and combining them, one search pass isn't enough. Two passes (strict filter + gentle top-up) with iterations deliver 1.5-2x better recall than single-pass chain of thought. And most importantly — compactness beats redundancy. 5 precise passages work better than 50 noisy ones, even if those 50 contain everything you need — because the model gets lost in a long context.

Context: The researchers gave an example of a multi-hop question from 2WikiMultiHopQA and showed how the Selector ⇔ Adder loop works. The question required finding two people and comparing their dates of death.

Question Analyzer prompt (English original):

You are given a complex question that requires retrieving multiple pieces of evidence
from different documents. Your task is to decompose this question into a list of
simpler sub-questions. Each sub-question should target a single fact needed to
answer the original question.

Question: {question}

Output Format:
1. [Sub-question 1]
2. [Sub-question 2]
...

Instructions:
- Be explicit about what information each sub-question seeks
- Make each sub-question self-contained
- Cover all reasoning hops required

Selector prompt (English original):

You are given a list of sub-questions and a set of candidate passages.
Your task is to SELECT ONLY those passages that DIRECTLY answer at least one
sub-question.

Rule: When in doubt, exclude. It is better to miss a passage than to include
irrelevant content.

Sub-questions:
{sub_questions}

Candidate Passages:
{numbered_passages}

Output: Comma-separated list of passage IDs that should be selected

Adder prompt (English original):

You are given a list of sub-questions, candidate passages, and a set of
ALREADY SELECTED passages.

Your task: Check if any sub-questions remain UNANSWERED by the selected set.
Add passages that COMPLEMENT the selection and fill gaps.

Rule: Do not duplicate already selected passages. Only add what is truly missing.

Sub-questions:
{sub_questions}

Already Selected:
{selected_passage_ids}

All Candidates:
{numbered_passages}

Output: Comma-separated list of NEW passage IDs to add (or "none" if complete)