Where the Task Sits in Your Context: Why Some LLMs Lose the Question Unless It Comes Last

PRINCIPLE: The task always goes at the end of the context

✅ CORRECT STRUCTURE:

{document / examples / background context}

────────────────────────────

Task: {question / instruction}

──────────────────────────────────────────────

❌ RISKY STRUCTURES:

Task: {question}       ← question first, context after (not tested in the paper)
{large context}

──────────────

{context 1}
Task: {question}       ← question in the middle
{context 2}

──────────────────────────────────────────────

🔧 DIAGNOSTIC TRICK (if the question has to sit in the middle):

{context 1}
Task: {question}       ← question in the middle
{context 2}

────────────────────────────

Reminder: solve this task: {question}  ← copy at the end

It all works inside a single prompt. No separate requests.

Our illustration, not an example from the paper (the paper used math and science test questions).

Task: You're reviewing a commercial lease for retail space in Chicago, 15 pages of text. You want to know whether any clauses could hurt your business.

❌ What most people do (risky structure):

Review this lease and flag every clause 
that could be unfavorable to the tenant.

[15 pages of lease...]

The question sits far from the point where the model starts answering, with a mountain of text in between. Note that the paper only compared a task in the middle with a task at the end; it did not test the question-first layout, so treat this as a likely risk rather than a measured one.

✅ The right structure:

[15 pages of the lease...]

────────────────────────────────────────

Your task: flag every clause in the lease above 
that could be unfavorable to the tenant. 
For each clause, explain what the risk is 
and what's worth negotiating with the landlord.

Expected result: The model gets a clear task after the context, the position where models in the study did best. You should get a focused list of risks rather than a generic recap of the lease.

🔧 If the structure forces the question into the middle (say, documents and instructions are interleaved):

[Lease, part 1: pages 1-8...]

Pay special attention to the rent escalation clauses.

[Lease, part 2: pages 9-15...]

────────────────────────────────────────

Task reminder: flag every clause in the lease above 
that could be unfavorable to the tenant, 
especially anything about rent escalation.

The LLM weakness: A common explanation is that a model reading a long context gives extra weight to the most recent tokens, which sit closest to where the output begins. The paper itself does not pin down the mechanism: it shows the effect is about position, and leaves the cause open.

What goes wrong with a task in the middle: If the question is surrounded by lots of similar content (solved examples, other questions), the model "grabs" answers from nearby text instead of solving the task. The study found that, in the first five models tested, 76% of wrong answers at the middle position matched the answer of a neighboring example problem. At the end position, such matches (mostly coincidences between small numbers) account for only 22%.

Why repeating it at the end works: When a copy of the task comes last, the model "sees" it right before it starts generating. At 8K tokens that wiped out the positional effect: repetition restored accuracy to within ±4 percentage points of the "ideal" position for all 9 models. At 64K it was only partly effective (see Limitations).

Control levers:

{background_context}

────────────────────────────────────────

{task_or_question}

What to fill in: - {background_context}: the document, article, data, email thread, examples, anything you want the model to take into account - {task_or_question}: the specific instruction or question, always last

If the context comes in parts:

{context_part_1}

{context_part_2}

────────────────────────────────────────

Task: {question_or_instruction}

If the task has to sit in the middle:

{context_part_1}

{intermediate_instruction}

{context_part_2}

────────────────────────────────────────

Reminder, your task is: {question_or_instruction}

⚠️ Model dependence: The effect varies a lot across models. DeepSeek in reasoning mode barely loses accuracy with similar background content, but it does drop with unanswered background questions. The newer releases (GLM-5.1, Kimi-K2.6, DeepSeek-V4-Pro) hold up much better than older versions when the background is solved examples, but still lose 16 to 56 points when it is unanswered questions. Your model may behave differently, so test it.

⚠️ The type of background content is critical: Neutral text (Wikipedia, news) causes almost no interference for most models, with one exception: Qwen 2.5-7B still lost about 30 points at 32K and 64K. Similar examples are the danger, especially if they contain numbers or answers of the same type as your task. Don't use solved examples as "separators" when the task sits in the middle.

⚠️ The effect grows with context length: For most vulnerable models the drop at 8K tokens is moderate (Qwen 2.5-7B is the exception: already -86 points at 8K). At 64K it's catastrophic for vulnerable models. Contexts shorter than 8K were not tested. If you work with big documents, the "task last" rule goes from nice-to-have to mandatory.

⚠️ The diagnostic trick isn't always 100%: With very long contexts (64K), repeating the task at the end fully restored accuracy for only 2 of the 4 models checked (Qwen 2.5-7B reached 84% vs 94%, GLM-4.7-FlashX 78% vs 90%), and the newer releases were not checked at 64K. At 8K it works for all 9 models. The authors call it a diagnostic, not a proven fix.

Researchers from Beijing Jiaotong University took 9 current LLMs from different providers (DeepSeek, Kimi, GLM, Qwen, MiMo) and tested them on math word problems (GSM8K) and science questions (ARC-Challenge). The idea was simple: take the same task, surround it with background text, and see what happens when the task sits at the end versus in the middle across different context sizes: 8K, 32K and 64K tokens.

The key experiment was a "diagnostic probe": they placed a task in the middle (where it loses accuracy), then added a copy of the same task at the end. At 8K, accuracy bounced back to the "task at the end all along" level for all 9 models. The authors read this as consistent with the issue being position, not task difficulty, while noting the probe does not isolate the exact mechanism.

The most surprising result: different kinds of background content have radically different effects. At 64K, MiMo-v2-Flash loses 88 percentage points of accuracy when the background is solved problems, and nothing at all (it scores 2 points higher) when it's neutral Wikipedia text. So for this model it's not "long context kills accuracy": it's topically similar content that creates interference (Qwen 2.5-7B, by contrast, dropped even with neutral text). Most wrong middle-position answers match an answer from the surrounding examples, which suggests the model is pulling answers from them instead of solving your task.

A telling detail from the audit: the researchers checked the release reports of four major vendors (DeepSeek, Kimi, GLM, MiMo) and found that none of them included position-controlled tests such as NIAH, RULER or LongBench in their main result tables. The headline tables showcase coding and agent benchmarks. Where such tests appear at all, it is outside the main tables (for example, in a base-model table or an architecture ablation).

Results table: accuracy drop when the task moves from the end to the middle (GSM8K, background = solved problems):

Model               | 8K Drop | 32K Drop | 64K Drop
────────────────────|────────|─────────|─────────
Qwen 2.5-7B         |  −86pp★ |  −84pp★  |  −94pp ★
MiMo-v2-Flash       |  −12pp  |  −24pp   |  −88pp ★
MiMo-V2.5-Pro       |   −8pp  |  −16pp   |  −32pp ★
GLM-4.7-FlashX      |  −12pp  |  −20pp   |  −34pp ★
GLM-5.1             |   +0pp  |   +2pp   |   +2pp
DeepSeek-R          |   +0pp  |   +0pp   |   +0pp
DeepSeek-V4-Pro     |   −2pp  |   −2pp   |   −4pp
Kimi k2.5           |   +4pp  |   +0pp   |   −6pp
Kimi-K2.6           |   +2pp  |   −2pp   |   +2pp

★ = statistically significant (Bonferroni-corrected); other drops are not

Diagnostic probe (8K, each model on its worst background type):

Model           |  End  |  Middle  | Middle×2  |  Middle+End
──────────────  |──────|─────────|───────────|────────────────
Qwen 2.5-7B     |  90%  |    4%    |     4%    |      94%
MiMo-v2-Flash   |  98%  |   22%    |    70%    |      98%
MiMo-V2.5-Pro   | 100%  |   44%    |    68%    |     100%
Kimi k2.5       |  94%  |   40%    |    92%    |      96%
DeepSeek-V4-Pro | 100%  |   68%    |    72%    |      96%

Context: The researchers tested 9 LLMs through the providers' official APIs (DashScope, Xiaomi, Zhipu, DeepSeek, Moonshot). N=50 examples per condition, seed=42.

1. Adaptation: working with multiple documents

When you need to compare several documents or feed the model many sources, the structure is the same:

💡 Adaptation for comparing documents:

Copy

=== Document 1 ===
{text 1}

=== Document 2 ===
{text 2}

=== Document 3 ===
{text 3}

────────────────────────────────────────

Compare these three documents on the following criteria: {criteria}

2. Technique: a "closing reminder" for long conversations

🔧 Technique: restate the task at the end of a long conversation

If your chat with the model has stretched across many messages and answer quality is slipping, restate the task explicitly in your next message:

Copy

[Quick reminder of what we're working on: we're {task_summary}]

Next step: {specific_instruction}

Same logic, extended by us: "task last" should help at the level of the whole conversation history too. The paper only tested single prompts.

3. Extrapolation: stress-test your own model

Want to see how vulnerable your model is to the positional effect? Run a quick test:

Copy

Take any question with a known answer.
Put the question in the middle of 500-1000 words 
of similar questions (neutral text rarely triggers the effect).
Compare the answer with the version where the question comes last.

If the answers differ, your model is vulnerable 
to the positional effect even with short contexts.

Paper title: Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks

Authors: Chuyifei Zhang, Hongyu Cui, Xiaowen Huang, Jitao Sang

Universities: Beijing Jiaotong University, Central South University of Forestry and Technology

Benchmarks mentioned: Needle-in-a-Haystack, RULER, GSM8K, ARC-Challenge, SCROLLS, LongBench, LooGLE, BABILong