The Anatomy of Long Context: Where Models Lose Information and Why "Don't Make It Up" Kills Accuracy

1. Positional effects: where to put what matters

Lost in the middle isn't a myth — it's an architectural trait of some models:

• ChatGPT-5-mini: fails at exactly 50% context depth
• Claude-4.5-haiku: a U-shaped curve — the middle (20-60%) collapses
• Gemini-2.5-flash and Deepseek-v3.2-chat: stable at every position

Practical takeaway: If you work with ChatGPT or Claude — don't put critical information in the middle of a long prompt. Place it at the beginning or the end.

2. Effective length vs maximum length

Models accept more tokens than they can use effectively:

Practical takeaway: If you use ChatGPT or Claude for long documents — don't fill them to the limit. Better to split into parts or pick Gemini/Deepseek for tasks with truly long contexts.

3. The safety tax: the price of "Don't Make It Up"

The prompt "Don't make things up if you don't know" reduces hallucinations but causes over-refusal (the model won't answer even when the information is there):

ChatGPT-5-mini (at maximum length): - Fact extraction: 89% → 72% (-17%) - Logical inference: 88% → 68% (-20%) - Hallucinations: 73% → 90% (+17% faithfulness)

Deepseek-v3.2-chat: - Fact extraction: 99% → 97% (-2%) - Logical inference: 92% → 94% (+2%) - Hallucinations: 84% → 87% (+3% faithfulness)

Practical takeaway: - For ChatGPT/Claude: use "Don't Make It Up" only if faithfulness is critical and you're ready to lose 15-20% of correct answers - For Deepseek/Gemini: use it freely, the trade-off is minimal

4. How information is distributed matters more than length

The researchers tested different distributions of facts across the context (normal, exponential, uniform). Models suffer not from context length but from sparse facts.

When 10 facts are scattered across 200k tokens, even strong models start missing them. When the same facts are concentrated in one block, extraction is nearly perfect.

Practical takeaway: If you paste a long document into a prompt, group the relevant information — don't rely on the model's ability to find a needle in a haystack. Either pre-extract the pieces you need (manually or with a separate request), or use RAG.

Task: Pull the key takeaways from a long podcast transcript

You have a 2-hour podcast transcript — an interview with serial entrepreneur Mark Davis (hypothetically, ~150k tokens). You need to extract every concrete piece of business advice Davis mentions in the conversation.

❌ Bad approach (information gets lost):

Here's a podcast transcript. Find all the business advice.

[the entire 150k-token transcript]

Problems: - The advice is scattered throughout the text - The model may miss facts in the middle (lost in the middle) - ChatGPT will start degrading at this length

✅ Good approach (using the study's findings):

Option 1: Structured extraction in two steps

Step 1. Read the transcript and find ALL the passages where Mark Davis gives business advice, shares his experience or criticizes other people's approaches. 

List the timestamps where this happens in the format:
- 05:23 — on hiring
- 12:45 — on competitors
...

If you're not sure — include it anyway. Don't skip anything.

[transcript with timestamps]

Step 2. Here is the list of passages that contain advice. Extract ALL of these passages from the transcript and phrase the advice in the format:

**Takeaway:** [a short statement]
**Quote:** [what Davis said, word for word]
**Timestamp:** [minutes]

[list from step 1]
[transcript]

Option 2: With Deepseek/Gemini you can do it in one prompt

You are an analyst who studies the business thinking of successful entrepreneurs.

Task: Extract ALL the business advice from the transcript. 

Critically important:
1. DO NOT SKIP advice from the middle of the conversation — read the whole text carefully
2. Include concrete actions, principles, and criticism of other people's approaches
3. If the advice is implicit — infer it logically from the context

Output format:
**[Timestamp]** — **[Short takeaway]**
Quote: "..."
Context: [why it matters / what it relates to]

[transcript]

Result:

With the two-step approach or with Deepseek/Gemini, you'll get a complete list of advice with quotes and timestamps. With ChatGPT/Claude on a long transcript without structuring, there's a high risk of missing advice from the middle of the conversation (the 20-60% stretch).

LLMs use an attention mechanism that distributes "cognitive resources" across tokens. In long contexts, attention gets diluted — each token receives less weight.

An architectural quirk: models were trained on texts where important information is usually at the beginning (introduction) or the end (conclusions). The middle is development, details, transitions. So the attention pattern is skewed toward the edges of the context. It's not a bug — it's the result of training on real-world texts.

"Don't Make It Up" works like a confidence threshold. The model generates an answer with some probability. Add "don't make things up" and the threshold rises. But in models like ChatGPT this threshold becomes too high: the model refuses to answer even when the information is there, just not in the top attention positions (the middle of the context, scattered facts).

Deepseek and Gemini are better calibrated: their confidence threshold more accurately reflects whether the information is really there. ChatGPT and Claude are more conservative — their safety tax is higher.

Levers you control:

1. Position of information → Put critical content at the beginning or end
2. Context length → Don't fill it to the limit; leave 30-40% headroom
3. Anti-hallucination instructions → Use them only if faithfulness is critical, and understand the trade-off
4. Model choice → For long contexts: Deepseek/Gemini. For short ones, ChatGPT/Claude are fine
5. Structuring information → Group what's relevant; don't scatter facts across the whole text

Based on the study's findings, here's a universal template for working with long context:

{a clear instruction — what to extract / analyze}

{critical constraints and requirements — if any}

{output format — structured}

IMPORTANT: {if completeness matters — say explicitly "read the whole document, don't skip the middle"}

{if faithfulness is critical — add: "If the information isn't there, say so. Don't make it up."}

---

{DOCUMENT / CONTEXT}

---

{repeat the key instructions — to exploit recency bias}

Key principles:

1. Sandwich structure: instruction BEFORE the document + repeat AFTER

   • Recency bias: the model remembers what came last best
   • Primacy bias: the model remembers what came first best
   • Use both

2. Visual separators: ---, ===, ### between sections

   • Help the model tell instructions apart from data
   • Especially important for long contexts

3. Explicit completeness instructions:

   • "Read the ENTIRE document carefully"
   • "DO NOT SKIP information from the middle"
   • This calibrates the model's attention

4. Structured output:

   • Tables, lists, numbering
   • Easier to check for completeness
   • Harder for the model to skip an item

5. Anti-hallucination — selectively:

   • Add it only if faithfulness is critical
   • Understand you'll lose 5-20% accuracy (depending on the model)
   • For Deepseek/Gemini the trade-off is minimal

Based on the study's results:

For tasks with long contexts (100k+ tokens):

✅ Gemini-2.5-flash: - Stable up to 1M tokens - Minimal safety tax - No positional bias - Best choice for: analyzing large documents, long transcripts, multiple files

✅ Deepseek-v3.2-chat: - Stable up to 128k tokens - Excellent anti-hallucination calibration - No positional bias - Best choice for: tasks where both completeness and faithfulness matter

For tasks with moderate contexts (up to 50k tokens):

✅ ChatGPT-5-mini: - Works great up to 100k - But a high safety tax - Fails at 50% depth - Use when: the context is short, the information is at the beginning/end, and faithfulness isn't critical

⚠️ Claude-4.5-haiku: - U-shaped dip (lost in the middle) - Degrades past 100k - Use with caution: only for short contexts with the information at the beginning/end

⚠️ Study specifics: It was tested on literary narrative (Balzac's novels). Structured documents (tables, code, JSON) may behave differently — attention patterns there may not be the same.

⚠️ Models get updated: The results are for specific versions (GPT-5-mini, Claude-4.5-haiku, etc). Future versions may handle long contexts better. The principles hold, but the specific numbers may change.

⚠️ RAG wasn't tested: The study is about pure long context. For real tasks with very long documents, RAG (Retrieval-Augmented Generation) may be more effective — first find the relevant pieces, then analyze them. But that requires setting up a system; it doesn't work "out of the box" in a chat.

⚠️ The safety tax depends on the task: The accuracy drop from "Don't Make It Up" is more pronounced at maximum context length and with scattered facts. On short prompts with compact information the effect is weaker.

The team ran an extended needle-in-a-haystack test, but much harder than the standard version. They took 38 novels by Balzac (~2M tokens) as the "haystack" — a coherent narrative, harder than random text. They planted 30 "needle" facts and asked questions about them.

Two protocols:

1. Uniform sweep: 200 tests per model — varying the context length (10-100% of the model's maximum) and the fact's position (10-100% depth). This produced a failure map: exactly where the model loses information.

2. Probabilistic distributions: 18 tests per model — scattering 10 facts across the context according to 9 statistical distributions (normal, exponential, etc). This simulated real documents, where information isn't all in one place.

Each test was run in two prompting modes: - Standard: "Answer the questions about the text" - Anti-Hallucination: + "Don't make things up; if you don't know, say you don't know"

They measured three metrics: - Literal Extraction — the model found the fact "as is" - Logical Inference — the model drew a conclusion from several facts - Faithfulness — the model didn't fabricate (100% = no hallucinations)

All models were tested with the same hyperparameters (temperature=0, top_p=1.0) for reproducibility. Answers were checked by an independent LLM judge against a clear answer key.

Why the results came out this way:

They found three failure patterns:

1. U-shaped memory (Claude): The attention architecture is skewed toward the edges of the context — that's how models learned from real texts, where the important stuff is usually at the beginning/end.

2. 50%-cliff (ChatGPT): A specific vulnerability at exactly the halfway point of the context — possibly an artifact of the architecture or training.

3. Conservative failure (all models with anti-hallucination): "Don't Make It Up" raises the confidence threshold. If a fact sits in the middle of the context (low attention weight) → the model isn't sure → it refuses to answer. In ChatGPT/Claude this threshold is too high; in Deepseek/Gemini it's better calibrated.

Surprisingly: Context length by itself isn't the main problem. Gemini handles 1M tokens, Deepseek 128k. The problem is how attention is distributed across that length. When facts are scattered, even a short context hurts; when they're grouped, a long one doesn't get in the way.

The practical insight: Don't ask "does the model support N tokens" — ask "at what length does it reliably extract information from positions 20-60%". That's the effective length.

Not All Needles Are Found: How Fact Distribution and "Don't Make It Up" Prompts Shape Literal Extraction, Logical Inference, and Hallucination Risks in Long-Context LLMs

Amirali Ebrahimzadeh (University of Michigan), Seyed M. Salili (Independent Researcher)

The study tested Gemini-2.5-flash, ChatGPT-5-mini, Claude-4.5-haiku and Deepseek-v3.2-chat on an extended needle-in-a-haystack benchmark with a corpus of novels by Honoré de Balzac.