Multi-step prompt chains (where you ask an LLM to do a task in stages: first parse the data, then calculate, then phrase the answer) lose accuracy sharply if each next step sees only the output of the previous one rather than the original task. The researchers call this loss the "decomposition tax" — and measure it directly: same model, same prompts, the only difference being whether each step remembers the original question.
The gap is huge: a model solving the task in one go does several times better than the version split into stages — the loss reaches 40 accuracy points (in one case, from 55% down to 15%). The reason is simple: when an intermediate step "retells" the task in its own words or writes the data out as a list, it loses some of the links between conditions — like a game of telephone. The next step works with an impoverished version and can't notice, because it never saw the original.
The fix is "re-grounding": at the right step in the chain, show the model the full original task again, not just the previous step's output. This recovers most of the lost accuracy. If you can insert the original only once, insert it after the problem step, not before it. And if an intermediate step compresses data into a list, explicitly ask it to preserve the relationships between quantities rather than just listing them.
Method Blueprint
WITHOUT REPAIR:
Task → Step 1 (compresses/processes) → output
→ Step 2 sees ONLY Step 1's output → ...
→ final answer
(accuracy drops sharply at the seams)
WITH REPAIR (re-grounding):
Task → Step 1 → output
→ Step 2 receives: [Step 1 output] + [reminder: "Original task (for reference): ..."]
→ ...
→ final answer
If Step 1 compresses data into a list — add to its instruction:
"...and preserve all relationships stated between quantities"
Everything happens within one chain of chat messages — no extra calls, just edits to the prompt text.
Example in Action
Task: You're putting together a home remodel estimate for a client through a chain of prompts in a chat — first you ask the LLM to pull every cost item out of your email thread with the contractor, then to total the amounts by category, then to write up the estimate for the client.
Prompt (a chain of three messages):
Message 1:
Here is the thread with the contractor: {thread text}
List all the amounts, materials and work items mentioned.
Preserve all the relationships between them — what goes with what,
who pays for what, which amounts depend on others.
Message 2 (after the bottleneck — with re-grounding):
Original thread (for reference): {thread text}
Result of the previous step: {output from message 1}
Calculate the total for each expense category
(materials, labor, delivery) and the grand total.
Message 3:
Based on the calculations above, put together an estimate for the client
as a clear table with explanations.
Result: The final estimate will be more accurate and won't be missing cost items — because the second step sees not only the "cleaned-up" list from the first step but also the original thread, which may contain nuances (discounts, conditions, one amount tied to another) that the first step could have dropped while compressing.
Why It Works
When an LLM retells or compresses information at an intermediate step, it inevitably loses some details and links — that's an unavoidable property of retelling, not a bug in a particular model. The next step in the chain doesn't see the original and can't notice — it simply trusts the trimmed-down version.
An LLM's strength is that it works well when it sees the whole picture at once. One comprehensive request is almost always more accurate than a chain of several trimmed-down handoffs.
Re-grounding leverages that strength: instead of making the model trust someone else's retelling, you give it the original context again at the right point. It's like reminding a person of the original assignment, not just a coworker's summary of it.
Control levers: - Where to insert the task reminder (before the problem step or after) → it's more effective after: inserting it before the bottleneck can even be worse than no repair at all. - How the intermediate step is phrased (list vs. text, with an explicit "preserve the relationships" requirement) → strongly affects how much information gets lost. - Number of steps in the chain → the more "opaque" handoffs between stages, the bigger the cumulative loss — cut the number of stages wherever you can.
Prompt Template
[Message 1]
Here is the task: {task_text}
{stage_1_instruction}. Be sure to preserve all relationships
stated between the elements of the task — don't just list them separately.
[Message 2 — after the stage where data was compressed]
Original task (for reference): {task_text}
Result of the previous step: {stage_1_output}
{stage_2_instruction}
[Message 3 — final]
{final step instruction: formulate the answer based on everything above}
🚀 Quick start — paste into a chat:
Here is a template for handling a multi-step task with a prompt chain.
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask how many stages you have and where in the chain information gets "compressed" (a list, a short summary) — because it's right after such a compressing stage that you need to remind the model of the original task.
Limitations
⚠️ Token cost: repeating the original task at each step increases the amount of text in the prompt. If the task is a long document, it eats noticeably into the context.
⚠️ Not a complete cure: even with the repair, the chain almost never catches up with the accuracy of a single comprehensive request covering the whole task. The median recovery is about 63% of what was lost, not 100%.
⚠️ If the task can be solved with one prompt — don't split it: the method fixes a chain but doesn't make it better than a monolithic request. Only split into stages when you really need to (a long process, different roles, a limit on answer length).
How They Studied It
The researchers took 21 open models (from 1 to 24 billion parameters, from nine different developers) and ran them through hand-built 3–4-step chains on math problems (GSM-Hard and MATH-500), 200 problems per configuration. They compared two modes: "strict" (each step sees only the previous step's output) and "full" (each step is additionally shown the original task) — the accuracy gap between them is the "decomposition tax."
To make sure text length wasn't the cause, they added a placebo control: inserting extra text with no real task content — it didn't restore accuracy, proving the issue is lost meaning, not the volume of the prompt. Surprisingly, even the newest 2026-generation models keep losing accuracy in chains — the loss mechanism differs a bit (older models depend more on the data handoff format, newer ones on whether the next step is allowed to reinterpret the data), but the effect itself and the way to fix it remain the same. The main practical takeaway: the hope that "a newer model is smarter and won't lose context in a chain" doesn't pan out — you have to fix it by explicitly repeating the task.
Resources
The Decomposition Tax: LLM Pipelines Lose Up to 40 Accuracy Points at Their Own Interfaces. Preprint, under review at ICLR 2027. Authors: Tianqi Bu (Rutgers University), YuXuan Peng (Nanjing University of Posts and Telecommunications), Junteng Tu (New York University), Henghui Xiao (Rutgers University / Nanjing Tech University).