The Consistency Filter: How to Tell a Model's "Confidence" from the Truth in Entity Extraction

STEP 1: Extract with roles → list of entities + role (core / background) + quote
STEP 2: Repeat 5 times (separate requests/chats) → 5 lists
STEP 3: For each entity, compute the share of runs where it appeared (0.2 … 1.0)
STEP 4: Add how often it's mentioned in the text (except for "techniques"/descriptive terms)
STEP 5: Sort → the "top" items per entity type go to review first
STEP 6: A human checks the context; confidence = priority, not truth

Steps 1–2 are model requests. Steps 3–5 are a manual tally or a simple spreadsheet.

Task: An analyst at an investment firm gets a 60-page report on the Warner Bros. Discovery split. They need a list of the companies that are actually party to the deal. The text has dozens of mentions: JPMorgan, Citi, Revolut, Amazon, Meta — as background, for comparison and in footnotes.

Prompt:

You are extracting entities from a report. Read the document below.

Task: find the companies that are PARTICIPANTS in the deal splitting up
Warner Bros. Discovery (buyers, sellers, vehicles the deal runs through).

Rules:
1. For each company, give its role: "participant" or "background"
   (mentioned for comparison, as a competitor, in footnotes, in history).
2. For each company, give a verbatim quote from the document
   that shows its role.
3. If you're not sure whether it's a participant or background, mark it "background" and explain.
4. Do not give a confidence percentage.

Format: table | Company | Role | Quote |

Document:
{report_text}

Run this prompt 5 times in fresh chats. Then merge the tables: how many times did each company get the "participant" role?

Result: The model returns a table of companies, roles and quotes. After 5 runs a picture emerges: some companies are consistently "participants" (5 out of 5), some flicker between "participant" and "background," and some are "background" every time. Check the flickering ones first. Don't blindly trust the 5-out-of-5 ones either: if one false statement made it into the report, they may be resting on it.

The weakness. The model pulls out strings; it doesn't decide whether they belong to this event. A self-reported "95 confidence" is just a habitual answer: nearly all scores bunch up in the 80–100 range. So there's nothing to rank by.

The strength. The model is good at finding mentions (80–99% recall). And if an entity is shaky, it shows up as spread across runs. When the text is ambiguous, the model's "core or background" call varies from run to run, and that disagreement is measurable.

How the method uses this. It doesn't ask the model to grade itself. It looks at how reproducible the answer is. Strong repetition means "stable given the text," not "correct." The authors tested this: they added one sentence with a false claim of involvement to a report, and in 95–100% of cases the models "confidently" tied the false entity to the event. Stability measures consistency, not fact.

Control knobs: - Number of runs (5) — fewer gives coarse steps; more sharpens the ranking but costs more. - Entity type — for group and organization names, mention frequency in the text is a strong signal; for descriptive terms ("techniques") it hurts. - Review budget — for identifiers like ID numbers, checking the top ~20% of the list is enough; for organizations you need to check everything in the top group. - No self-assessment — drop "confidence in %" from the prompt; it doesn't help and creates a false sense of reliability.

You are extracting entities from a document.

Document: {document_type}
Main event / subject of the document: {event}
What to extract: {entity_type}

Rules:
1. Extract only entities that relate to the main event.
2. For each entity, give its role:
   - "core" — directly involved in the event;
   - "related" — connected to the event, but not central;
   - "background" — mentioned for comparison, history, as an example or in a footnote.
3. For each entity, give a verbatim quote from the document.
4. If you're torn between "core" and "background," choose "background" and explain why.
5. Do not rate your confidence with a number.

Format: | Entity | Role | Quote |

Document:
{text}

Run it 5 times in separate chats. Then ask:

Here are 5 tables. For each entity, count in how many runs
it got the role "core", "related", "background". Output a table
sorted by the number of runs with the role "core".

What to fill in: {event} — the document's subject in one sentence; {entity_type} — companies, people, ID numbers, terms; {text} — the full document.

Caveat: the paper doesn't include the authors' actual prompt text. The template above is a practical implementation of their findings: separate core from background, don't ask for self-assessment, count repeats over 5 runs.

🚀 Quick start — paste into the chat:

Here's an entity-extraction template that separates "core / background" and uses five runs.
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about the document type, the event and the entities — because the line between "core" and "background" depends on what counts as the center of the document. Without that definition, the model will dump everything into one list again.

⚠️ Confidence is not truth: a single false sentence in a document makes the model "confidently" accept it almost every time, and even a sentence explicitly flagged as "unrelated to the case" still got a false entity attached in a noticeable share of cases. Repeatability doesn't protect you from bad input.

⚠️ The top of the organization list is unreliable: for groups/organizations, the top of the ranking turned out to be packed with errors (11 of the first 13 were wrong attributions). You have to manually check the whole top, not just the middle.

⚠️ One domain: tested on cybersecurity reports and three entity types. For contracts, news or résumés, you'll have to find a useful threshold all over again.

⚠️ Thresholds don't transfer: on another corpus the ranking stayed useful, but fixed cutoffs ("≥0.8 — trust it") didn't. They have to be recalculated for each document set.

⚠️ The fix isn't universal: adding retrieval and step-by-step reasoning helped only for organizations and didn't help for identifiers; on descriptive terms the effect flipped sign between models.

⚠️ Error analysis was partly done by LLMs: the error causes were labeled by two annotator models, not humans. The authors disclose this openly; agreement is good but not perfect. The counterfactual test wasn't run on the flagship model.

⚠️ You need a manual tally or a script: counting repeats across 5 runs by hand in a chat is doable, but across hundreds of documents you'll need automation.

The researchers took a public set of 354 campaigns (315 reports) on cyberattacks. The task was to pull out threat group names, vulnerability IDs and attack techniques. They tested two cheap new-generation models plus one flagship model as a control, 630 runs in total. The scoring rules and thresholds were locked in in advance, before the experiments, so the results couldn't be massaged.

First, it turned out that precision was low (3–34%) with recall at 81–99%: the model finds almost everything it should but drowns in extras (on average 15 IDs per report versus at most two in the gold standard). The flagship did no better. Then 360 erroneous candidates were analyzed by hand (more precisely, by two annotator models from other companies). The result was a surprise: nearly two thirds of the extras were background mentions, a fifth were "event" entities outside the gold standard, and only 1.2% were fabrications. So the thing to fix isn't "hallucinations" — it's understanding context.

Next they compared confidence signals: a 0–100 self-report, the share of repeats across 5 runs, and how often the word appears in the text. Repeats cleared the pre-set threshold in all six "model × type" combinations. The self-report added almost no information on top of them. Finally, they inserted one false sentence into each of 45 reports: an attribution, a denial, "unrelated" background. Confidence didn't budge, hence the conclusion: it ranks, but it doesn't verify.

🔧 Technique: "background" as a separate role → cuts off the main source of errors

Add a third role to the prompt and the rule "when in doubt, mark it background." This hits the main cause of excess output head-on (65.6% of errors). Without that role, the model dumps everything it sees into the answer.

🔧 Technique: drop self-assessment, add a quote → review gets fast

Instead of "confidence 0–100," require a verbatim quote. The reviewer only has to read one line of context. In the paper, the link between the quote and the value also turned out to be weak (quote overlap with the source mostly below 0.5), so still eyeball the quote.

Extrapolation (the authors didn't test this, it's an idea):

Ask a second model not to "confirm" the extraction, but to answer just one question about each "flickering" entity: "Does it relate to the main event, or is it mentioned as background? Give a quote." The authors showed that generic self-checking ("is this correct?") often confuses "mentioned" with "involved," so a narrow question about role rather than truth is closer to what worked for them.

• Paper: From Unreliable to Usable: Decomposing Failure Modes and Confidence Signals in LLM Entity Extraction — Evidence from Cyber Threat Intelligence
• Author: Jiahong Qian, Shanghai Jian Qiao University, Shanghai
• Built on: the Di Tizio et al. (2022) corpus and the critical audit by Mezzi et al. (2025a), whose data splits were used for comparability
• Related: Xiong et al. (2024), Guo et al. (2017) on poor calibration of verbalized confidence; Wang et al. (2023) on self-consistency