How to reduce LLM hallucinations with prompts

Symptom: fluent answers with invented parts

The model states a law, a date, a citation or a statistic in the same calm tone it uses for things it actually knows. The error is not visible in the text: the author is real, the journal is real, the year is plausible, but the paper itself was never written. In agent workflows the same failure looks different: a tool returns an empty result, and the assistant reports an order status anyway.

Three variants are worth telling apart, because they need different fixes: invented facts from memory (closed-book questions), drift away from a document you supplied (summaries, contract questions), and invented data in multi-step agent runs.

Why it happens

The studies converge on one mechanism: the model is built to produce a plausible continuation, and nothing in a plain prompt rewards it for stopping.

• Answering always pays. The authors of I-CALM argue that in training an answer of any kind was rewarded, while "I don't know" earned nothing, so a guess is the default.
• The request itself triggers fabrication. In the citation audit, no model invented a single reference when references were not asked for; asking for them switched the model into generating citation-shaped strings.
• Gaps get filled with what is typical. A single line asking to consider diversity raised invented patient demographics from 0.7% to 33% of answers, because the model pulls the usual co-occurring details into the case.
• Silent empty data. When a tool reports success but hands back nothing, fabrication rates spike to 45%; when the tool returns an explicit error, the model stays honest.
• The question gets swapped. Trace Inversion shows the model often answers a convenient rephrasing of your question with full confidence.

Fixes ranked by measured effect

The fixes below are ordered by the size of the effect the authors reported, on their own models and tasks. The metrics differ (accuracy of kept answers, F1 of error detection, fabrication rate), so treat the order as a rough guide, not a league table.

1. Keep only what repeats. Real facts reproduce; inventions do not. In the citation study, references named by one model matched a real paper 16.5% of the time, those named by three or more models 95.6%; within one model, references that appeared in at least two of three runs matched 88.9%.
2. Give the model a named failure state. A single line, retrieval_status: OK/FAILED, before the answer cut fabrication by 16x in tool-using agents.
3. Ask the same thing two ways and compare. In Disagreement-Based Abstention a direct question is compared with a decomposed version; for Gemini Pro on Mintaka a correct answer was 50 times more likely when the two agreed, and the method beat "Are you sure?", "say I don't know" and self-consistency on F1 in 16 of 18 model-dataset pairs.
4. Check in explicit steps, not with one question. HalluClean (plan, execute, verdict, fix) raised detection F1 from 33% to 68% on GPT-3.5; the task router line alone took summarization from 25% to 60%.
5. Select, do not rewrite. For questions about a document, SafeLLM numbers the lines and asks only for line numbers, then rebuilds the answer from the source; the study reports up to 95% agreement on medical terms.
6. Reward abstention in the prompt. With scoring rules (+1 correct, −1 wrong, +0.4 for "I don't know") plus honesty principles, I-CALM lowered false answers for GPT-5 mini from 52.3% to 34.2%.
7. Make the model list what is missing. The evidence-sufficiency wrapper (data available, data missing, is it sufficient, answer) reduced confident ungrounded answers by 38% from structure alone.
8. Ask for what is well anchored. Asking for "seminal and foundational" instead of "recent and influential" sources lowered the hallucination rate from 74.1% to 55.0% in the citation audit.
9. Separate recall from formatting. REAP recalls facts in free text, slice by slice, and formats JSON in a later step; the paper reports more complete lists but gives no single number.

A combined prompt stack

The fixes stack because they act at different points: before the answer, inside it, and after it.

1. Before: ground the input. Annotate raw values with what is normal and what is a fault, as in the epistemic stability study, and remove hidden premises ("find X if any exist, then rank them").
2. Inside: put the I-CALM scoring rules and the line "Do not make assumptions without evidence" at the top, and for agents require the status line before any content.
3. Inside, for documents: "Use ONLY the information from the provided text", plus line selection when wording must stay exact.
4. After: run the four-step check on anything that will be published, and for citations or key figures repeat the request in fresh chats and keep the overlap.

Each layer costs tokens or extra requests, so match the stack to the cost of an error.

What does not work

• "Are you sure?" The model usually says yes. In the disagreement study this baseline reached F1 of 0.51-0.76 on Llama-70B and Qwen-8B, against 0.88-0.94 for comparing two prompt modes.
• Asking it to be more careful. An instruction to double-check the data cut deception by only a third.
• Harsher penalties. Raising the error penalty to −100 barely changed behavior in I-CALM; the reward for abstaining is the lever.
• Reasoning under impossible constraints. When GPT-4o had to return "journal articles only" where none fit, reasoning cut open violations from 75% to 13% but raised disguised metadata forgeries from 3% to 11%.
• Too much caution. In the sufficiency study one model's accuracy fell from 75% to 17% once it started refusing almost everything; soften to "answer with an explicit caveat" if that happens.

You will be scored according to the following rules:
- Correct answer: +1
- Incorrect answer: -1
- "I don't know": +0.4

Principles:
1. Always tell the truth (do not lie or deceive).
2. Treat all topics with respect and humility.
3. Do not make assumptions without evidence.
4. Be responsible for your words and their consequences.
5. Honor reason, clarity, and understanding.

Question: {question}

Please respond as follows:
- If you know the answer, provide it along with your confidence score (between 0 and 1).
- If you are not sure, say "I don't know" first, then provide your best guess along with your confidence score (between 0 and 1).

You are checking {content_type} for factual errors and hallucinations.

{CONTENT_TYPE} TO CHECK:
{text_to_check}

STEP 1: Draft a verification plan. Which facts need to be verified? What could the sources of error be?

STEP 2: Carry out the plan step by step. Check each fact and show your reasoning.

STEP 3: Draw a conclusion: does the text contain hallucinations? Yes or No.

STEP 4: If yes, rewrite the text with the errors fixed.

── REQUEST 1 ──
Number every line of the text below.
Numbering format: [N] line text
Number every line, including short ones.

{paste the document or section text}

── REQUEST 2 (after you get the numbered text) ──
Here is the numbered text.
Question: {your question}

Instructions:
- Return only the numbers of the lines needed to answer
- Pick the minimal sufficient set
- No text of your own — numbers only
- Format: comma-separated numbers

{paste the numbered text from request 1}