TL;DR
LLMs hallucinate references not on their own, but in response to a request: without an explicit ask for sources, not one of 10 models produced a single citation. That means the problem isn't inside the model, it's in how you ask. Researchers at Clemson University checked 69,557 citations from 10 commercial LLMs against three academic databases and found two practical filters that sharply boost accuracy.
The main pain point: you ask an LLM for sources and get something that's plausibly formatted but doesn't exist. The author is real, the journal is real, the year is real, but that paper was never written. Spotting it without checking by hand is nearly impossible. The fabrication rate ranges from 11% for the best models to 57% for the worst. The reason: models "remember" the pattern of a citation, not the source itself. When there's no real anchor in the training data, they generate something plausible rather than something true.
Two filters solve this in practice. First: ask three different LLMs the same thing and keep only what all of them named; accuracy jumps from 16% to 96%. Second: repeat the request three times in one model and keep the references that showed up at least twice; accuracy hits 89%. Both filters work because real sources are stable and fabricated ones are random.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
Two independent filters. Use them separately or together.
Filter 1 — Multiple models (3 separate chats):
STEP 1: Ask ChatGPT the same question about sources → list of references
STEP 2: Same request in Claude → list of references
STEP 3: Same request in Gemini (or any third model) → list of references
STEP 4: Keep only the sources that appeared in ≥2 answers
→ prioritize the ones that appeared in 3 out of 3
Filter 2 — Repetition in one model (3 requests in one chat or in fresh chats):
STEP 1: Send the request → reference list #1
STEP 2: Same request again → reference list #2
STEP 3: Same request again → reference list #3
STEP 4: Keep only the references that appeared ≥2 times out of 3
Bonus filter — How you phrase the request (one prompt):
Before: "give me recent and influential sources on topic X"
After: "give me seminal and foundational sources on topic X"
That alone cuts hallucinations by 19%, automatically.
Example in Practice
Task: Emma Miller is writing an analytical brief for an investment committee and needs the key academic papers on how social media affects political polarization. She asks ChatGPT for a list of sources, pastes it into her slides, and a week later her colleagues find three papers that don't exist.
A prompt with the phrasing filter + repetition:
What are the seminal and foundational academic papers
that studied the impact of social media on political polarization?
Give me a list: authors, title, journal, year.
Only papers that actually exist — if you're not sure, leave it out.
Run it 3 times in a row (a new chat or a new request), then:
Here are three lists of sources on [insert topic].
Pick out only the papers that appear in at least two of the lists.
List 1: [paste]
List 2: [paste]
List 3: [paste]
Result: The model gives you the overlap: a short list of 5-10 sources that came up consistently. These are your candidates for real papers. You still need a final check in Google Scholar or Semantic Scholar, but the list shrinks from "50 questionable" to "8 probably real."
Why It Works
An LLM doesn't "remember" papers; it "predicts" plausible text. When you ask for sources, the model generates a string that matches the pattern of a citation: surname, title, journal, year. If a real paper showed up many times in its training data, it will reproduce it accurately. If not, it will generate something that "sounds about right." The two outcomes look identical.
Real sources are stable; fabricated ones are random. That's the study's key insight. A fake citation is a stochastic artifact: it appears once in a specific context and is unlikely to show up again. A real paper exists as a "center of gravity" across many documents in the training data, so the model reproduces it consistently. The repetition filter exploits exactly this difference.
The phrasing "seminal/foundational" reduces hallucinations because those papers are cited thousands of times and firmly anchored in the model's data. "Recent and influential" asks the model to find something that's either missing from its data (past the cutoff date) or barely there. So the model "fills the gap" with fiction.
Knobs you can turn: - Number of repetitions → 2 out of 3 is a good balance. If the topic is very narrow, raise it to 3 out of 3 - Number of models → 3 models give 96% accuracy. Two models already get you 87%, also useful - Phrasing → "seminal / foundational / classic / fundamental" instead of "new / recent / latest" - Domain → the less popular the field (structural engineering, niche medical topics), the higher the risk; use both filters
Prompt Template
Template 1 — Change the phrasing
What are the seminal and foundational academic papers
that studied {topic}?
Give me a list: authors, title, journal or conference, year.
Include only papers you're confident actually exist.
If you don't know the exact title, leave it out.
{topic} — your field: "the microbiome's effect on immunity," "machine learning in finance," "managing teams under uncertainty"
Template 2 — Overlap filter (after 3 repetitions)
Below are three lists of sources on {topic}, obtained independently.
Pick out only the papers that appear in at least {2 or 3} of the three lists.
For each selected paper, give: how many times it appears + full details.
List 1:
{list_1}
List 2:
{list_2}
List 3:
{list_3}
{topic} — the topic of your request
{2 or 3} — 2 for balance, 3 for maximum caution
{list_1/2/3} — the answers from the three previous requests
🚀 Quick start — paste this into the chat:
Here are templates for checking LLM-provided sources for hallucinations.
Adapt them to my task: {your task — for example,
"I'm writing a literature review on neural networks in medicine"}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask about your topic and how many sources you need, so it can frame the base request for the repetitions properly. It will take the pattern from the template and generate three versions of the request for you to run one after another.
Limitations
⚠️ The filter doesn't guarantee 100%: Even when three models agree, ~4% of sources are still hallucinated. A final check in Google Scholar, Semantic Scholar or JSTOR is mandatory.
⚠️ Narrow and niche topics are a danger zone: Structural engineering, regional studies, rare medical fields: hallucinations reach 50%+. The narrower the topic, the stricter your checks need to be.
⚠️ "Newer" doesn't mean "better" for citations: haiku-4.5 hallucinates 57% of the time vs. 49% for haiku-3.5; the newer model turned out worse. Don't count on "the latest version" as a guarantee of accuracy.
⚠️ Models favor popular, open papers: Even the real sources LLMs name are mostly highly cited open-access papers. Lesser-known but important work is likely to be skipped, or fabricated "in its spirit."
⚠️ Changing the phrasing lowers the risk but doesn't remove it: "Seminal sources" still get hallucinated, just less often. The trick cuts errors by ~19 percentage points but doesn't fully solve the problem.
How the Research Was Done
The Clemson University team ran what may be the largest experiment on citation hallucinations to date. 10 commercial LLMs from OpenAI, Anthropic, Meta, DeepSeek, Mistral and Moonshot AI got identical prompts across four academic fields and produced a total of 69,557 citations, which were automatically checked against three databases (CrossRef, OpenAlex, Semantic Scholar).
A clever design detail: a control condition with no request for sources, i.e. the same topic questions without the words "give me references." All 10 models, all 3,030 answers: zero citations. That proves hallucinations aren't "built into" the model; they're triggered by the request itself.
Another finding that surprised them: haiku-4.5 did worse than haiku-3.5 (57% vs. 49% hallucinations). That breaks the "newer = better" intuition: when picking a model for work with sources, the version number guarantees nothing. GPT-5-mini had the best result (11.4%), but generated fewer citations overall; maybe the model "knows what it doesn't know" and holds back sources when it isn't sure.
The most practical insight came from analyzing the overlaps: the researchers noticed that real sources are reproduced similarly by different models, while fabricated ones aren't. That turned into two verification filters, tested on the full dataset.
Original from the Paper (overlap check data)
Cross-model convergence results:
- Cited by 1 model: match rate 16.5%
- Cited by 2 models: match rate 87.4%
- Cited by ≥3 models: match rate 95.6% (5.8× improvement over single-model)
Within-model repetition results:
- Appears in 1 of 3 replications: match rate 28.6%
- Appears in ≥2 of 3 replications: match rate 88.9% (3.1× improvement)
Framing effect (across all 10 models, no exceptions):
- "Recent and influential" framing: hallucination rate 74.1%
- "Seminal and foundational" framing: hallucination rate 55.0%
- Difference: 19.1 percentage points
Context: The authors checked every citation through an automated pipeline with fuzzy matching on titles (Jaccard distance), authors and year. They also manually validated a sample of 225 citations using GPT-4.1-mini with web search; accuracy on the confirmed ones was 100%.
Adaptations and Extrapolations
💡 The stability principle applies beyond citations
The same mechanism works for any factual data: statistics, dates, names, technical specs. If an LLM gives the same number three times, it's probably real. If it's different every time, that's your cue to check.
🔧 Technique: apply the repetition filter to facts
I'm going to ask you the same question three times. After all three answers, pick out only the facts you stated in all three answers. This is my check for how stable the information is. Question: {factual question} [After the answer]: Now answer the same question again, without looking at your previous answer. [Repeat three times, then ask for the overlap]
💡 Adapting it for business research
Writing a competitive analysis or market overview and asking an LLM to name the key players, market data or studies? Same risk. Ask for "established, well-known market players" instead of "new and promising startups."
💡 Extrapolation: "old vs. new" framing as a general principle
The study shows that the more firmly a fact is "anchored" in the training data (mentioned a lot, been around a long time, widely available), the more accurately the model reproduces it. Which means:
- Ask LLMs to explain classic concepts, not to formulate new definitions
- Use LLMs for well-known technologies, not for describing brand-new tools
- When you need fresh information, ask for a structure and evaluation criteria, not the facts themselves
Resources
Title: How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations
Data and code: - HuggingFace: huggingface.co/datasets/mznaser/llm-citation-hallucination-audit - GitHub: github.com/mznaser-clemson/llm-citation-hallucination-audit
Author: M.Z. Naser, PhD, PE — School of Civil and Environmental Engineering & Earth Sciences, Clemson University; Artificial Intelligence Research Institute for Science and Engineering, Clemson University, USA. Contact: [email protected]
Related research cited in the paper: Walters & Wilder (the baseline GPT-3.5/4 study), Linardon et al. (the topic-familiarity factor), TruthfulQA, HalluLens (ACL 2025)