Inference-Aware Judge: A Content-Checking Prompt That Doesn't Penalize Expert Inference

Everything runs in a single prompt for the LLM judge:

INPUT: [Data source] + [Document being checked]
       ↓
STEP 1: Extract all facts from the source → list F
STEP 2: Extract all claims from the document → list C
STEP 3: For each claim → apply the 5-step protocol (below)
       ↓
OUTPUT: List of claims with a verdict: Supported / Hallucination + reason

Five-tier classification (built into the prompt):

Five-step protocol (run for each claim):

1. State the claim
2. Search the source: direct mention, synonym, paraphrase, professional equivalent
3. Found → SUPPORTED. Stop.
4. Not found → ask: "Is this a reasonable inference from the context or a standard summary?"
   Yes → SUPPORTED. Stop.
5. Otherwise → HALLUCINATION. State the reason.

Hard floor (never counts as supported, under any conditions): - Specific numbers, dates or names that aren't in the source - A conclusion that directly contradicts the facts in the source

Task: A strategy consultant ran a two-hour session with the team of a startup backed by a Sequoia-level fund. An assistant wrote up a meeting summary with conclusions and recommendations. You need to check whether everything in the summary is supported, or whether some facts were made up.

Prompt:

You are a document verification expert. You are given a meeting TRANSCRIPT and a SUMMARY 
written from it. Check every claim in the summary.

TRANSCRIPT:
---
[Paste the meeting transcript]
---

SUMMARY:
---
[Paste the summary]
---

VERIFICATION RULES:

A claim is SUPPORTED if any of the following holds:
- It was stated directly in the transcript
- It is a professional rephrasing or synonym of what was said
- It is a reasonable inference that follows from the facts taken together
- It is a standard takeaway for this type of meeting, consistent with the context

A claim is a HALLUCINATION only if:
- It introduces information with no basis whatsoever in the transcript
- It directly contradicts what was said

HARD RULE: if a claim names specific figures, names, deadlines or 
commitments, they must appear in the transcript verbatim.

For each claim in the summary, apply the protocol:
1. State the claim
2. Search the transcript: direct mention, synonym, semantic equivalent
3. Found → SUPPORTED. Stop.
4. Not found → "Is this a reasonable inference from the context?" Yes → SUPPORTED. Stop.
5. Otherwise → HALLUCINATION. Reason.

Check facts against ALL parts of the transcript, not just the matching section of the summary.

Result: The model returns a breakdown of every claim in the summary with a verdict and an explanation. The consultant's conclusions ("the team isn't ready for a Series A") will be marked as supported if they follow from the facts of the meeting — even if nobody actually said the words "Series A." Made-up figures or commitments get flagged as hallucinations, with a note on exactly what's wrong.

When asked to "check," an LLM defaults to looking for an exact text match. That's not a bug — it's a consequence of how generation works: the model is conservative under uncertainty and picks the safest answer. Without explicit criteria for "supported," it treats any mismatch as a risk.

The problem is that real work with documents is interpretation, not transcription. Consultants, analysts and lawyers don't copy the source — they draw conclusions. "Revenue has fallen three quarters in a row + customer churn is rising" → "the company is in crisis" isn't a hallucination, it's an inference. A standard checking prompt doesn't get that.

Control levers: - Add tier 2a/2b examples — give the model 2–3 "original → professional translation" pairs right in the prompt. It gets sharper at recognizing terminology equivalents in your field - Raise the hard floor — add to the forbidden list whatever matters in your context: budgets, names, dates, commitments - Remove the "Stop" — let the model always walk through all 5 steps → you see the full chain of reasoning for each claim - Specialize the role — instead of "document verification expert," give a specific one: "seasoned investment analyst" → better tuned to the domain

You are an experienced {role: analyst / lawyer / consultant / editor} with expertise in {domain}.

You are given a SOURCE and a DOCUMENT based on it.
Check every claim in the DOCUMENT.

SOURCE:
---
{source text: transcript / data / material / interview}
---

DOCUMENT:
---
{text being checked: summary / report / brief / analysis}
---

VERIFICATION RULES:

A claim is SUPPORTED if:
- It is directly contained in the source
- It is a synonym, paraphrase or professional translation of what was said
- It is a reasonable inference that {an expert in {domain}} would draw from the facts
- It is a standard concluding statement consistent with the overall picture

A claim is a HALLUCINATION only if:
- It introduces information with no basis whatsoever in the source
- It directly contradicts what is in the source

HARD RULE: {specific figures / names / deadlines / amounts} must appear 
in the source verbatim.

For each claim in the DOCUMENT:
1. State the claim
2. Search the source: direct mention, synonym, semantic equivalent
3. Found → SUPPORTED. Stop.
4. Not found → "Reasonable inference or standard conclusion?" Yes → SUPPORTED. Stop.
5. Otherwise → HALLUCINATION. State the reason.

Check claims against the entire source, not just the adjacent section.
Hallucination is the verdict of last resort, not a first reaction.

Placeholders: - {role} — who is checking: analyst, lawyer, editor, product manager - {domain} — subject area: investing, marketing, medicine, law - {source text} — the data source: transcript, data, interview - {text being checked} — what you're checking: summary, report, brief - {specific figures / names / deadlines} — what needs literal accuracy in your context

🚀 Quick start — paste into the chat:

Here is an Inference-Aware Judge template for checking documents. 
Adapt it to my task: [describe what you're checking and what it's based on].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about the domain, the role and what exactly needs checking — because that determines how precisely "reasonable inference" can be defined for your field. It will keep the protocol from the template and adapt it to your task.

⚠️ Needs domain-specific context: The better you describe the role and domain, the more accurately the model draws the line between "reasonable inference" and "speculation." Without it, the result is still better than literal checking, but worse than the full version.

⚠️ Doesn't replace expert review for critical decisions: The method cuts false positives but doesn't guarantee zero errors. For legal and medical documents, the final verdict belongs to a human expert.

⚠️ Works worse on subjective judgments: If there's no objective criterion for "reasonable inference," the model will be too lenient. The method is for factual checking, not for evaluating style or tone.

⚠️ Long documents need to be split: With very large sources (>20,000 characters), cross-section checking quality drops because of context window limits.

The LLM's weakness: Under uncertainty, the model picks the conservative answer. "No direct match" → it's cheaper to answer "not supported" than to reason about whether it's an inference. Without instructions, the model behaves like a box-ticking inspector, not an expert.

The LLM's strength: The model understands semantic links, synonyms and cause-and-effect chains well — if you explicitly ask it to apply them. It has the domain knowledge it needs; the problem isn't knowledge, it's instructions on how to apply it.

How the method uses this: Explicitly describing five tiers of support switches the model from "looking for an exact match" mode into "applying expert judgment" mode. The five-step protocol with "hallucination as the last verdict" removes the default conservatism. The result: the model works like an expert reviewer, not like grep over a text file.

The Augnito team collected 100 anonymized doctor-patient transcripts across specialties — cardiology, dermatology, endocrinology, ENT. For each transcript an LLM generated a SOAP note, and then another LLM evaluated it for hallucinations.

The idea was simple: run two versions of the judge — literal (Stage 1) and expert (Stage 2) — and compare them with ratings from real physicians. The literal judge got a standard prompt: "flag everything that isn't in the transcript." The expert judge got a prompt with explicit inference criteria and access to medical reference sources (SNOMED CT, ICD-10).

The result surprised even the authors: the literal judge found 35% hallucinations, the physicians found 10%. The expert judge landed at 9% — practically matching the humans. The 25-percentage-point gap isn't real model error, it's an artifact of a bad checking prompt. The most interesting part: both versions of the judge agreed on tiers 4 and 5 (true fabrications and contradictions). The whole difference lies in tier 3, clinical inference, which the literal judge methodically wiped out.

Stage 1 (literal checking):

"You are a medical fact checker. For each claim in the SOAP note, 
determine whether it is explicitly supported by the patient–physician 
conversation transcript. If the claim contains information that was not 
directly stated in the conversation, mark it as HALLUCINATED."

Stage 2 (expert checking):

"You are a clinical documentation expert and medical AI evaluator. 
When assessing SOAP note claims against the source transcript, apply 
the following criteria: A claim is SUPPORTED if it is (a) directly stated 
in the transcript, (b) a paraphrase or medical terminology equivalent of 
stated information, or (c) a medically reasonable inference that a trained 
clinician would draw from the presented symptom picture. A claim is 
HALLUCINATED only if it (a) introduces information with no basis in the 
transcript and cannot be reasonably inferred, or (b) directly contradicts 
information stated in the transcript."

Context: The researchers tested both versions on 100 doctor-patient transcripts. Stage 1 is the baseline prompt with no inference criteria. Stage 2 is a reworked prompt with explicit rules + retrieval from a medical knowledge base (SNOMED CT, ICD-10) for every claim being checked.

💡 Adaptation for checking marketing briefs

A brief written from a client briefing call. Classic situation: the client said one thing, the copywriter "built on it," and now it's unclear whether that's interpretation or invention.

You are an experienced strategist with expertise in brand communications.

You are given a TRANSCRIPT of a client briefing and a BRIEF written from it.

[Transcript and Brief]

VERIFICATION RULES:

SUPPORTED if:
- The client said it directly
- It is a marketing phrasing of the same meaning the client expressed
- It is a reasonable inference from the positioning, target audience or goals the client described
- It is standard phrasing for this type of task, consistent with the overall context

HALLUCINATION only if:
- It attributes to the client a tone, values or audience that were never mentioned
- It directly contradicts what the client said

HARD RULE: specific budgets, dates, KPIs and competitor names — 
only if the client named them personally.

[Five-step protocol]

🔧 Technique: replace "expert in X" with a specific persona → sharper judgment

Instead of:

You are an experienced analyst with expertise in investing

Try:

You are Warren Buffett, who has read thousands of investment memos 
and learned to tell well-founded optimism from marketing fog

A specific persona with character gives the model a sharper angle when judging Tier 3 — what counts as a "reasonable inference" and what is speculation. A faceless "expert" is softer.

🔧 Technique: add "explain the inference chain" for borderline cases

After step 4, add:

If you classify a claim as a "reasonable inference," 
state in one sentence which facts from the source lead to it.

This makes the check auditable: not just a verdict, but the logic behind it. Handy when you need to show the results to a client or your team.

Beyond Literal Summarization: Redefining Hallucination for Medical SOAP Note Evaluation

Augnito Research, April 2026

Authors: Bhavik Vachhani, Kush Shrivastava, Pranshu Nema, Sai Chiranthan

Medical expertise: Dr. Nikhil Saldanha

Knowledge bases used: SNOMED CT, ICD-10