Coverage Certificate: Make the LLM Explicitly Check Data Completeness Before Answering "No"

Coverage Certificate is a technique that makes the model, before concluding "definitely not," spell out two things: what exactly the question covers and what exactly the provided data covers. Only after comparing these two scopes does the model give its final verdict: "definitely not" or "I don't know."

Models almost always say "no" too confidently when they don't find something in a list or document, even when that list is incomplete. The reason is simple: the model silently decides "I didn't find it → so it doesn't exist," without checking whether it was shown all the relevant material. This goes especially badly when the incompleteness isn't stated outright and has to be inferred from context; that's where the model most often slides into false confidence.

The method turns this hidden (implicit) completeness judgment into an explicit one: the model names the scope of the question separately, the scope of the data separately, and only then compares them and decides. One fuzzy conclusion becomes three clear text fields.

Method Blueprint

STEP 1: Define the scope of the question → a text description (the whole period? the whole collection? all sections?)
STEP 2: Define the scope of the data → a text description + a label: complete / partial
STEP 3: Compare → does the data cover the full scope of the question? (yes/no)
STEP 4: Final conclusion → "Definitely not" (if yes) / "I don't know" (if no)

All four steps run in one prompt; the model returns the whole structured answer at once.

Example in Action

Task: A business owner received a 12-page supply agreement from a vendor and wants to check whether it contains a clause letting the supplier change the price unilaterally.

Prompt:

Here is the text of the agreement:
{paste the agreement text}

Question: does this agreement contain a condition giving the supplier the right
to change the supply price unilaterally?

Before answering, take three steps explicitly:

1. Scope of the question: determine what exactly I'm asking about: the whole document, 
   including exhibits and amendments, or just the main text?

2. Scope of the data provided: assess what exactly you were given. 
   Is it the whole document or only part of it (for example, without exhibits, 
   without specifications, without amendments)? Mark it: complete or partial coverage.

3. Comparison: does the text you were given cover the full scope of the question?

Give your final answer in this format:
- Scope of the question: ...
- Scope of the data: ...
- Conclusion: [There is definitely no such condition / I can't say for sure: insufficient data]

Result: the model returns three text fields instead of a plain "no, there's no such clause." If the exhibits or amendments to the agreement didn't make it into the prompt, the model will say so explicitly in the "Scope of the data" field and honestly answer "I can't say for sure" instead of jumping to the conclusion that the condition doesn't exist.

Why It Works

By default the model doesn't explicitly check whether it was shown all the material; it makes that call "in the background" and often slides into the simplified logic of "didn't find it = doesn't exist." That's the weakness: a hidden completeness judgment is easy to get wrong, especially when the gap isn't labeled with plain words like "this is an incomplete list."

The LLM's strength is that it breaks a task down into explicit text categories well when you ask it to directly. It can describe in words "what the question covers" and "what the data covers" separately, if you give it a structure for that.

The method leans on this ability: instead of one fuzzy decision, three explicit text fields force the model to describe completeness first and only then decide.

Control knobs: - Add reasoning (CoT) before the certificate: in the study, the "certificate + step-by-step reasoning" combo gave the best result. - Replace the final "I don't know" option with "insufficient data, I need the full document": more practical for real document work. - Drop the "no explanations" requirement: that way you can see where the model went wrong, in defining the scope of the question or the scope of the data.

Prompt Template

Here is the data: {document_or_list}

Question: {is X present in this data}

Before answering, take three steps explicitly:

1. Scope of the question: determine what exactly is being asked: the whole period, 
   the whole collection, all sections, or only part?

2. Scope of the data: assess whether you were given everything or only part. 
   Mark it explicitly: complete or partial coverage (and why).

3. Comparison: does the data cover the full scope of the question?

Give your final answer in this format:
- Scope of the question: ...
- Scope of the data: ...
- Conclusion: [Definitely absent / I can't say for sure: insufficient data]

In {document_or_list} put the text, file or list you're checking. In {is X present in this data} put the specific fact whose presence you're checking (a contract clause, an entry in a registry, a condition in a list of exclusions).

🚀 Quick start: paste this into the chat:

Here is a Coverage Certificate template for checking that a fact is absent from a document.
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask which document and which fact you're checking, because without that it can't fill in the "scope of the question" and "scope of the data" fields correctly. It will take the template's structure and adapt it to your case.

Limitations

⚠️ The problem doesn't disappear, it shifts: explicit prompts and structured certificates reduce false confidence in "no," but at the same time increase unnecessary "I don't know" answers. There's no perfect prompt that removes both errors at once.

⚠️ Hidden incompleteness is the method's weak spot: when the data is incomplete but that isn't stated outright (only visible from context), the model gets confused most often, even with the certificate.

⚠️ Not for high-stakes decisions without human review: for legal, medical and financial conclusions the method lowers the risk but doesn't guarantee correctness; you need an extra check.

How They Tested It

The team built the CROWN-QA benchmark in two parts. The first is synthetic question pairs where the question and the observed facts are held fixed and only one thing changes: how much of the requested scope the provided data covers. The second is contrast sets on real documents: ACL Anthology conference indexes and DailyMed drug labels.

They tested three models (Qwen, Gemma, Claude Haiku) on seven different prompting strategies, from a plain question to explicit rules, reasoning and structured certificates. They measured the rate of false "no" answers (over-closure) and the rate of unnecessary "I don't know" answers (under-closure).

The main surprise: without special instructions, models almost always lean toward saying "no" too confidently, not the other way around. And even the best prompts don't eliminate the problem; they just redistribute the errors between the two types. The structured certificate helped with diagnosis: most often the model already goes wrong when describing "how complete the data is," not at the final decision.