Retrieve-then-Reason: How Context Length Kills Reasoning Even With Perfect Retrieval

Task: A 50-message thread with a client about launching an ad campaign. Somewhere in the middle they mentioned the budget, the channels and the timeline. You need to put together a proposal. When you paste the whole thread into a prompt with "write a proposal," the model gets confused and misses details — the long context gets in the way of reasoning, even though the information is right there.

Prompt (Step 1 — Extraction):

Here is my email thread with a client about an ad campaign:

[the whole thread, 50 messages]

Find and quote EXACTLY (word for word) every message where the client mentions:
- The campaign budget
- Ad channels
- Launch dates
- Creative requirements

Copy each relevant message in full. Don't make anything up, don't shorten anything.

Prompt (Step 2 — Solving on a short context):

Here's what the client said about the ad campaign:

[messages extracted in Step 1]

Based on this information, write a proposal.

Result:
In Step 1 the model returns 4-6 specific messages with the details. In Step 2, working on a short context, it accounts for every condition precisely and drafts the proposal without gaps. Instead of wandering through 50 messages, it focuses on a tight digest.

LLMs are good at extracting information from long text when the task is simply to find and quote. But when they have to hold a long context in "working memory" and solve a problem at the same time, performance drops.

The attention mechanism works like this: the longer the context, the more computation per token, and the weaker the link between the relevant parts. Even if the model "sees" all the evidence, it uses it worse when reasoning over long distances.

The method sidesteps this through a division of labor: first compression (pulling out what matters), then reasoning on the compressed data. Short context = less noise in attention, more precise work with the information. It's the difference between "find the needle in the haystack and solve a problem with it" vs. "here's the needle, solve the problem" — the second is easier, even though the needle is the same.

Prompt control knobs:

• Extraction instructions in Step 1: "quote exactly" vs. "paraphrase briefly" — the first for accuracy, the second for compression
• Completeness check: add "Make sure you didn't miss anything, reread the context" — insurance against gaps
• Format in Step 2: from "give the answer" to "lay out your reasoning step by step" — changes the depth of analysis

STEP 1 (first request — extraction):

{long_context}

Find and quote EXACTLY (word for word) every passage that mentions:
{search_criteria_list}

Copy each relevant passage in full. Don't make anything up, don't shorten anything.

---

STEP 2 (second request — solving):

Here is the relevant information:

{passages_extracted_in_step_1}

{your_question}

What to fill in:

• {long_context} — an email thread, document, log, any long text
• {search_criteria_list} — what to look for: dates, numbers, requirements, mentions of topics
• {passages_extracted_in_step_1} — the output of the first request, pasted here
• {your_question} — the final task: write a report, give a recommendation, make a decision

🚀 Quick start — paste into the chat:

Here's the Retrieve-then-Reason template. Adapt it to my task: [your task].
Ask me questions to fill in the fields.

[paste the template above]

The model will ask what your long context is (an email thread, a document, a log?), what exactly needs to be extracted, and what the final question is — because Step 1 needs clear search criteria and Step 2 needs a concrete task on the short data. It'll take the pattern and adapt it to your case.

⚠️ Depends on retrieval: If the model can't extract the information (a weak model, a very complex context, poorly worded criteria), the method won't help — the problem shows up as early as Step 1.

⚠️ Extra request: Doubles the number of API calls, adding latency and token costs.

⚠️ Risk of losing context: If Step 1 doesn't extract everything important, Step 2 works with incomplete data and may miss nuances.

⚠️ Doesn't fix retrieval: The method helps you use what was found, but doesn't improve the search itself — that's a separate problem.

The team took 5 models (Llama-3.1-8B, Mistral-7B, GPT-4o, Claude-3.5, Gemini-2.0) and tested them on math (GSM8K), questions (MMLU), code (HumanEval) and a synthetic task (Variable Summation). Each problem was stretched by inserting distracting tokens between the evidence and the question — from 0 to 30K tokens (Paul Graham essays as filler).

The key difference from other work: retrieval was measured separately via exact match — the model had to quote the evidence word for word. A single mismatched character meant 0 points. That's stricter than "got the gist but paraphrased." This made it possible to isolate the effect of retrieval: if the model quotes exactly, retrieval is perfect, and any drop in solving accuracy has a different cause.

The results were surprising: for Llama-3.1-8B at 30K tokens, retrieval fell by only 10%, but solving accuracy fell by 24%. In other words, the model sees the information (it can quote it) but can't use it effectively for reasoning!

Three control experiments to pin down the cause:

1. Replaced the essays with whitespace — minimal distraction. The drop got smaller, but was still up to -20% at 30K whitespace tokens. So it's not just about semantic noise.
2. Moved the evidence to the end, right before the question (instead of the beginning). Distance between evidence and question = 0. The drop remained. So it's not the "Lost-in-the-middle" effect.
3. Masked out ALL distractions — the model sees only the evidence and the question, identical to the short version, but separated by a 30K-token mask. The drop is still there: -21% on MMLU. So the culprit is length itself, not the content of the distractions.

Conclusion: Even under ideal conditions (exact retrieval, no distractions, close positioning), the model reasons worse in a long context. It's not a retrieval bug or a position issue — it's a fundamental limitation of working with long sequences.

Closed-source models are more stable: GPT-4o and Gemini-2.0 drop by 1-8% (vs. 20-30% for open-source), but they degrade too. An anomaly: Claude-3.5 on MMLU crashed by -67.6% at 30K whitespace tokens — this needs a closer look; it may be specific to the model or the dataset.

The Retrieve-then-Reason mitigation was tested on RULER (QA tasks over documents): on GPT-4o it gave +4% on QA2 with a 32K context (68.4% → 72.4%). Modest, but consistent across all lengths. With weaker baselines, the effect is bigger.

An example prompt for Step 1 (extraction) from the RULER experiments:

Read the question based on the given documents. Choose the documents most relevant
to the question and copy them fully WORD BY WORD.

The following are given documents.

{context}

Read the question. Choose the documents most relevant to the question and copy them
fully WORD BY WORD. Only give me the relevant documents and do not output any other
words. Do not make up documents. Do not repeat documents. Use "# Document <number>"
at the start of each document, then copy its full content.

Question:
{question}

Context: This is the exact version for the RULER QA tasks, where the model picks the relevant documents from a set and quotes them in full. Step 2 used the standard RULER prompt, "Answer based on given documents," with the documents extracted in Step 1 instead of the full set.

"Context Length Alone Hurts LLM Performance Despite Perfect Retrieval"
Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A Huerta, Hao Peng
University of Illinois at Urbana-Champaign, Amazon, USC, Argonne National Laboratory, Hebrew University, University of Chicago