TL;DR
When a long tool output lives in a file instead of the context, some models stop looking at the official report. Instead, they pull numbers from their own files that they created along the way. The fix is three sentences in the tool description and the system prompt: here is the authoritative source, it is already at the right precision, take the value from there, not from your own files.
The agent computed correctly but answered wrong. The tool's report prints the number exactly as needed: 2.751. The agent's own file has a longer version: 2.75065. The model truncates the tail instead of rounding, gets 2.750, and picks the neighboring wrong option. The logs make this plain: the agent obtains the correct value just as often as before. What gets lost is the "what am I answering from" step. In this setup, GPT-5.1 and GPT-5.2 never open the report in up to 60% of runs. Accuracy drops by 8–11 points. Claude barely suffers.
The method is simple: name the authoritative source and forbid answering from scratch files. Asking it to "round correctly" barely helps: models truncate anyway. What works is telling it where to take the value from. That recovered 6–8 points of accuracy for three GPT models. The alternative is to put the report straight into the context if it fits.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Decide which tool output is the reference (report, summary table, log)
STEP 2: Add 3 sentences to the tool description:
• values in [file] are authoritative and at the required precision
• take the value from the relevant section of [file]
• do not answer from your own intermediate files: values there are not rounded
STEP 3: Duplicate the same sentences in the system prompt / CLAUDE.md
STEP 4: If the report is short, put it straight into the context instead of a file
It's all plain text, in a single prompt or instructions file.
Example in Action
Task: You're setting up an agent in Cursor. It calculates a freelancer's quarterly estimated tax from a Chase business account export. The script calc_tax.py writes a detailed report.txt where every amount is rounded to the whole dollar by the rules. The agent also likes to save its own debug_dump.csv with cents and tenths of a cent. The client needs an answer like "estimated tax due this quarter: $4,218." But the agent grabs $4,217.68 from the dump, truncates it to $4,217, and reports a figure one dollar short.
Prompt (in the tool description and in the system instructions):
The calc_tax.py tool writes the final file report.txt.
Values in report.txt are authoritative; they are already rounded to the dollar
according to the tax calculation rules.
For your answer, take the amount from the relevant section of report.txt.
Do not answer from debug_dump.csv or any of your other intermediate files:
values there are not rounded, and rounding them yourself can produce
the neighboring, wrong amount.
In your answer, state the section of report.txt the number came from.
Result: The agent opens report.txt and finds the right section. The amount in the answer matches the official rounded figure. The dump with cents stays a working file and never makes it into the answer. The last line about citing the section is an add-on to the method; it's not in the paper (see "Adaptations").
Why It Works
The weakness. When the whole report sits in the context, it's the only source in front of the model. When it's in a file, it competes with other sources, including the agent's own files. The agent's files are shorter and more precise in terms of digits, so the model happily reads them. But the numbers in them aren't shaped to the answer format. The model truncates them instead of rounding, and the answer drifts to the neighboring option.
The strength. The model is good at following a clear instruction about where to get the data. Tell it "take it from the report" and it goes there: skipping the report drops from ~50–60% to 0–2%, and the share of correct answers when the value was seen rises to 99%.
How the method exploits this. Source selection was breaking on the model's side. The instruction makes that choice for it. Asking it to "round down/up correctly" changes nothing, because the reading behavior stays the same: the model still goes to its own files and still truncates.
Levers: - File and section name → the more specific, the more reliable. "The reference output" without a file name doesn't work: the instruction has to be tied to the name. - Explaining the reason ("values there are not rounded") → helps the model understand why the ban exists. - Placement → the authors put the text both in the tool description and in every call result. Duplicate it in the system prompt and the instructions file. - Output size → if the report fits in the context, show it in full, no instruction needed. That worked just as well and saved tokens.
Prompt Template
The {tool} tool returns {what_it_returns}.
Values in {reference_file} are authoritative: they are printed
at the required precision ({precision_format}).
For your answer, take the value from the relevant section of {reference_file}.
Do not answer from {draft_files}: values there are not rounded
or not converted to the required format, and you may end up with the neighboring,
wrong option.
What to fill in:
- {tool} — a script, test runner, simulator, API wrapper.
- {reference_file} — the report the final figures come from.
- {precision_format} — "to the dollar," "2 decimal places," "as a percentage."
- {draft_files} — exports, dumps, logs the agent creates on its own.
Limitations
⚠️ Model-dependent: Claude (Opus and Sonnet 4.6) shows practically no loss when working from a file, and GPT-5.4's net effect is also near zero. The effect is strongest for GPT-5.1 and GPT-5.2.
⚠️ "Round correctly" doesn't work: models still truncate even when explicitly told not to. Only pointing to a specific source helps.
⚠️ You need to know the reference: the instruction requires naming which output counts as canonical. If there's no reference, or it's not obvious, the method won't work.
⚠️ Tested on a single tool: a geochemical simulator with four-option questions where the options differ in the last digit. Transferring to other tasks makes sense, but the paper didn't test it.
⚠️ Not a cure-all: GPT-5.1 still falls short of the "report in context" level. That gap is no longer about rounding but about the calculation itself and other stages.
⚠️ Single control run: the instruction-vs-control comparison relies on one run per model. The main conclusion is backed by other comparisons too, but the numbers are approximate.
How They Studied It
The researchers took PHREEQC, a real geochemical simulator from the U.S. Geological Survey. Its reports run up to 847 thousand characters, and the output is fully deterministic, so every printed number can be checked. The test used 158 four-option questions whose answer keys were re-verified by running the simulator: of the original 200 questions, 42 turned out to be defective. The answer options differ in the last digit, so guessing is off the table; you have to compute the number.
They compared two ways of delivering the report: in full in the context (up to 100 thousand characters) or as a table of contents with line-by-line reading. They also tested keyword search. The models tested were Claude Opus 4.6, Sonnet 4.6, GPT-5.1, 5.2, 5.4, and Gemini 3 Flash. Gemini was excluded: almost half of its answers failed to parse.
The study's key move was re-running all 11,007 simulator calls from the logs. For each trajectory, three things became visible: whether the agent computed the needed value, whether that value reached it, and whether it answered correctly based on it. It turned out GPT-5.1 and GPT-5.2 compute just as often as before. The reading step is what breaks: the agent often doesn't open the report and takes the value from its own file with extra digits. Out of 35 "traps" (a value that rounds to the correct option but truncates to the neighboring one), GPT-5.2 picked the neighboring option in 26 cases, and Sonnet in just one.
Then came the intervention: three sentences pointing to the source. Skipping the report fell almost to zero, and answer correctness when the value was seen rose to ~99%. Accuracy went up 6–8 points for three GPT models. GPT-5.4 has a curious twist: its overall accuracy didn't change, because a gain in calculation masked a loss in reading. If you only look at the bottom line, you'll never see the problem.
Adaptations and Extrapolations
💡 Adaptation for the agent's instructions file (CLAUDE.md / AGENTS.md): The paper tested the instruction in the tool description and in the call result. The same principle can go into the project's persistent instructions.
## Source of truth for numbers Take final figures only from reports/summary.txt. Files in tmp/ and debug/ are working drafts; values there are not rounded. Do not use them when answering the user.
🔧 Technique: add a source citation → check it with your own eyes. Ask the agent to state, at the end of its answer, which file and section each number came from. The paper didn't test this, but it's a quick way to spot a source swap without digging through logs.
After every number in your answer, add: [file, section].
💡 Extrapolation (not tested in the paper): the same principle applies wherever the agent creates its own intermediate versions of data: CRM exports, ad performance reports, unit economics calculations. Name the official version and explain why the drafts are worse. Test it on your own model: Claude may show no effect.
Resources
- Paper: Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy
- Authors: Ke Zhang, Maziar Raissi (University of California, Riverside); Sahchit Chundur, Jerry Peprah Owusu, Mohammad Javad Abdolhosseini Qomi (University of California, Irvine)
- Tool: PHREEQC (USGS), Parkhurst & Appelo, 2013
- Related work: Sen et al. (2026) on delivering search results as files; Xie et al. (2026) on models not always using their context; Zhang et al. (2026b) on the question set.
- The authors give the verbatim text of the three sentences in Appendix B; it isn't in the excerpt we had.