Trace Inversion: Catching Hallucinations by Reconstructing the Question

The paradox: when an LLM hallucinates, it isn't "unsure." It confidently answers its own question, not yours. You ask one thing, it quietly rephrases it into a "convenient" version and reports back with total confidence. Trace Inversion lets you catch the swap — ask the model to reconstruct, from its own reasoning, what it actually answered, and compare that to the original. The trick: reasoning is a trace of interpretation. A swapped question leaves a fingerprint in the reasoning — and that's exactly where you catch it.

Three messages in a row in the same chat. First: ask your question with step-by-step reasoning — get an answer and the chain of steps. Second: ask the model to look only at its reasoning and state which question it was really answering. Third: compare both questions — the original and the reconstructed one. A big gap = the answer is unreliable; fix the question or double-check the facts. Models are good at comparing meanings — and that same strength exposes them.

An LLM hits ambiguity in a question — and fills it in. It quietly "patches" the request to fit what it knows and confidently answers the patched version. Standard checks look at answer confidence. But confidence is maxed out — the model did answer correctly, just not your question. The swap leaves a trace in the step-by-step reasoning — from it you can reconstruct what the model "heard". "Compare two questions by meaning" is an LLM's strong suit. The weakness gets caught by its own strength.

Legal, medical, financial and historical questions — wherever a mistake is costly. Especially useful when the question contains ambiguity, jurisdiction-specific details or a hidden premise the model might quietly replace. Not for routine tasks — three messages instead of one is too expensive when the stakes are low.

1. Answer with reasoning: ask your question and add "think step by step, show every step" — get the answer plus the reasoning chain. 2. Reconstruction: ask the model to look only at its reasoning — not your question — and write, in one sentence, which question it was really answering. 3. Comparison: put both questions side by side — the original and the reconstructed one. Ask it to find gaps in intent and content and give a verdict: how far can the first answer be trusted? 4. Optional — numeric signal: add "rate the divergence on a 1–10 scale" to step 3 — you get a concrete number instead of a vague "slightly different."

[BAD]: `What terms are mandatory in a convertible note agreement under New York law?` — you get a confident answer but have no idea whether it answers your question. [GOOD]: Step 1: `What terms are mandatory in a convertible note agreement under New York law? Think step by step and show every step of your reasoning.` Step 2: `Look only at your reasoning above. Not at the question I asked. Which question were you actually answering? Write it down precisely, in one sentence.` Step 3: `Compare two questions. MY QUESTION: "What terms are mandatory in a convertible note agreement under New York law?" THE QUESTION YOU ANSWERED: [paste from step 2]. Are there gaps in intent or content? How far can the first answer be trusted?` If the model answered about "generic loans" rather than the specifics of conversion, it will tell you so itself. Without this check, the swap goes unnoticed.