Retrieved but Not Read: search-enabled models answer from snippets and never open the pages

This isn't a single method. The study yields a set of tested rules, and here's what follows from them:

RULE 1: Give the model today's date in the prompt
RULE 2: Put the source text in the prompt itself (paste the page) → almost no outdated answers
RULE 3: If the model finds the page itself → require it to open it and quote the line with the value verbatim
RULE 4: In a search loop, repeat the original question after each opened page → recovers ~a quarter of failures
RULE 5: Specify a final answer line that includes a "CANNOT CONFIRM" option

All of this can be done in one chat or in an agent instruction. Rules 1 and 5 were part of the prompts in the study itself. Rules 2 and 4 were tested in separate experiments. Rule 3 ("open and quote") wasn't directly tested by the authors. It follows from the link between reading the page and answer quality, but that's an inference from correlation.

Task: A marketer at a fintech startup is preparing an investor pitch. A slide says "The Fed's target range for the federal funds rate is 4.25–4.50%." They need to quickly check whether that's out of date, without getting a confident "yes" based on a snippet.

Prompt:

Today is {date}. Check this claim:
"The Federal Reserve's target range for the federal funds rate is 4.25–4.50%."

Verification rules:
1. Find the primary source (federalreserve.gov or the official FOMC statement).
2. You MUST open the source page; don't answer from fragments in the search results.
3. Quote verbatim the line with the value and the date of the decision.
4. If you couldn't open the page, or the decision date is older than the latest meeting, don't confirm the claim.
5. Before concluding, go back to the original question: "is the claim true as of {date}?" and check it against the quote.

The last line of the answer must be exactly one of these three:
FINAL: TRUE
FINAL: FALSE (current value: ...)
FINAL: CANNOT CONFIRM

Result: The model first runs a search and opens the source page (if it has a tool for that). Then it gives a verbatim quote with the value and date and checks it against "today." At the end comes one of the three FINAL lines. If it couldn't open the page, you should get an honest "CANNOT CONFIRM," not a confident "true." Check the quote with your own eyes: open the link and make sure the line is there.

The weakness. The model saves steps. A snippet looks like a finished answer, so the model stops. If the snippet or its memory holds an old value, it confirms it. The researchers looked inside the models and saw something odd. Even when the page is in the prompt, the correct answer gains the lead midway through the computation, but in the final layers the old answer overtakes it. The new value is "noticed," but the old one isn't displaced.

The strength. If the source text is right in front of it, in the prompt itself, the model uses it far more reliably. It doesn't spend effort searching for and opening the page. In the experiment, two models confirmed an outdated claim 14 out of 50 and 11 out of 47 times when they had to find the page themselves. When the same page sat in the prompt, errors dropped to 0 out of 50 and 1 out of 50.

How a practitioner uses this. You remove the steps where the model slacks off: you put the page in the prompt yourself, or force it to open and quote it. Repeating the question after the page solves a different problem. In an agentic loop the question sits at the very beginning, followed by lots of page text, and the question gets "forgotten." Repeating it brings it back to the end.

Control knobs: - Date in the prompt → the model understands the value may have changed after training. - A "CANNOT CONFIRM" option → gives the model a legitimate way out instead of guessing. In the authors' test, a fixed answer-line format cut false agreement in the proprietary model by 17.5 points. - Requiring a verbatim quote → you can check it by hand quickly. Wrong quotes almost always accompanied outdated answers (81.8% vs 0.9%). - Pasting the page manually → the strongest knob when the fact is critical.

Today is {date}. {Question or claim to verify}.

Source (if I have one):
<source>
{page text or an excerpt from a document}
</source>

Rules:
1. Rely on the source above. If there's no source, find the primary source ({source type}) and you MUST open the page. Don't answer only from fragments in the search results.
2. Quote verbatim the line your answer is based on, and the page's publication date.
3. If the source date is older than {freshness_threshold}, warn that the data may be outdated.
4. Before the final answer, go back to the original question: "{short version of the question}" and check the answer against the quote.
5. If you couldn't verify it, don't guess.

The last line of the answer must be exactly:
FINAL: {option_1} / {option_2} / CANNOT CONFIRM

What to fill in: - {date}: today's date. - {source type}: for example, "the Fed's website," "the company's official site," "SEC EDGAR filings." - {freshness_threshold}: for example, "3 months." - {option_1}/{option_2}: for example, TRUE / FALSE.

If you don't have a source, delete the <source> block. If you do, paste the text into the prompt.

⚠️ The "open the page" rule wasn't directly tested: the authors showed that answers given without opening a page are wrong more often, but that's a correlation, not an intervention experiment. Requiring the model to open and quote is a logical conclusion, not a measured result. Test it on your own tasks.

⚠️ Pasting the page doesn't cure everything: errors drop but don't disappear. In the mechanistic part, even with the page in the prompt, some models kept choosing the old value. Still double-check critical facts with your own eyes.

⚠️ Repeating the question is a partial fix: it recovered about a quarter of failed checks, with huge variance across models (from a few percent to more than half). It's a supplement, not a solution.

⚠️ A narrow class of facts: they tested facts of the form "the value changed after the model was trained" (a player's club, a job title, a rate, regulations). The worst failures were in sports, where the answer sits in the first line of the page. For long documents like drug labels, the conclusion might be different.

⚠️ One proprietary model: live search was tested on a single lightweight model; the other models were open-weight and ran with controlled search. Flagship models may behave differently.

⚠️ Part of the paper is useless to practitioners: the layer analysis (logit lens, J-lens) requires access to the model's internals. It explains why this happens but doesn't give you a tool.

The idea was simple: find facts that changed after the model's training cutoff, so the model couldn't know them from memory. The authors collected 225 such facts from sports, finance, medicine (FDA), the SEC and other areas. Each has an old and a new value, so an outdated answer is easy to tell from a correct one.

First they tested live search on one proprietary model: in 53.6% of episodes it answered from snippets without opening a page. Then all the models (one proprietary and six open-weight configurations) got the same frozen list of ten results, in which the right page was always available. That way failures can't be blamed on bad results. The share of answers without opening a page varied widely: from 2.6% for gpt-oss-120b to 76% for qwen3-8b.

The main comparison: a snippet-based answer versus an answer with an opened and quoted page. Outdated answers: 6.4% vs 1.3%; wrong quotes: 6.4% vs 2.3%. In sports, the gap for open-weight models was 20.2% vs 2.1%. Wrong quotes almost always accompanied an outdated answer: 81.8% vs 0.9%.

Then they tested interventions. When the page was placed directly in the prompt, errors nearly vanished (for gemma-4-31b, from 14/50 to 0/50). Repeating the question after each page recovered 41 of 157 failed checks. The layer analysis showed that the model "recognizes" the new value, but the old one wins at the last step. That was the surprise: reading the evidence doesn't guarantee using it.

🔧 Technique: remove the "CANNOT CONFIRM" option → see where the model is guessing

If you're testing a model on your own facts (rather than trusting it), ask for an answer with no option to decline. Compare it with an answer where declining is allowed. The discrepancies reveal facts the model is confident about without grounds.

🔧 Technique: verbatim quote → a second chat checks only the quote

Open a new chat and give it only the page and the quote from the first answer, with the question: "Is this line on the page, and does it support the value {X}?" The researchers showed that wrong quotes are strongly linked to outdated answers. This is an extrapolation; the cross-check itself wasn't tested in the paper.

For an agent instruction file (CLAUDE.md, AGENTS.md, system prompt):

When checking facts that can change (job titles, prices, rates, versions, regulations):
- Don't answer from search-result snippets. Open the source page.
- Quote verbatim the line with the value and the date.
- After each opened page, restate the original question to yourself and check your answer.
- If you couldn't open the page, write "not confirmed," not "probably yes."

This adapts the paper's findings to an agent instruction. The authors didn't measure the effect of this exact wording, but it builds on the same two tested mechanisms: reading the page and repeating the question.

• Paper: Retrieved but Not Read: Understanding Why Search-Augmented LLMs Fail to Use Available Evidence
• Authors: Owen Walters, Lihu Chen, Francesca Toni, Department of Computing, Imperial College London
• Code and data: github.com/ojwwalters/retrieved-but-not-read
• Related work: Wallat et al. 2025 (citation correctness and faithfulness to the source), Kale 2026 (built-in web search in modern LLMs), Xie et al. 2024 (conflict between a model's memory and external data)