Incomplete LLM answers: how to stop silent omissions

Symptom: what is there is right, what is missing is invisible

You ask for all the requirements in a brief, every risk in a contract, every acceptable answer for a grader, or each issue a review mentions. The answer looks clean and confident. The problem is what is not there: a category never examined, a clause marked compliant without being read, a valid option nobody thought of. Unlike a hallucination, an omission leaves no trace to catch on review.

Studies agree this is the dominant failure in extraction work. In medical data extraction, GPT, Gemini and Grok were accurate about what they found but missed a lot (meta-analysis extraction benchmark). In multi-solution tasks, models reported about 85% confidence while recall sat at 10-20% (Reasoning Overconfidence). And LLM Council cites findings that 44-60% of genuinely dangerous answers are omissions rather than invented facts.

Why it happens

• Enumerating is a different task from judging. A model writes a list item by item and stops when recall runs dry, without seeing what it never thought of. It caught a planted extra item 6-7 times more often than its own omission (Judging is not enumerating).
• Attention locks onto the first path. After finding a few answers the model narrows its search and declares it is done; raising temperature did not help (cognitive rigidity).
• Broad requests pull the most typical items. "Extract all requirements" returns explicit technical specs and skips context, constraints and goals (PEGS).
• Too many decisions in one pass. With a long checklist, the model checks a fraction and marks the rest compliant by impression (sharding). On multi-label tasks, output gets shorter after the first 2-3 categories (CFD).
• A sufficient answer feels complete. Found three of five? The model reports three and stops, unless the format forces it to look for every field.

Fixes ranked by measured effect

1. Ask for a rule, then check candidates one at a time. Instead of "list every correct answer", have the model state the criterion, test it on a known-good example, then judge each candidate separately. Accuracy rose from about 0.48 to 0.99 across six model families on mechanically checkable tasks (rule instead of a list). Fuzzy criteria narrow the gap without closing it: model-written tests still rejected 58-81% of correct solutions.
2. Split the target into named categories. Four lenses (Project, Environment, Goals, System), each finished before the next, raised category coverage from 61% to 92% and F1 from 0.71 to 0.88 (PEGS prompting). The idea transfers: replace "all" with explicit angles.
3. Batch long checklists and require a quote. Checking a 100-item list in one pass verified only 15-30% of items; groups of 4-32 items, each with the full document and a supporting quote before the verdict, fixed it at the same total budget (sharding, Carnegie Mellon).
4. Give role, focus list and an output template. Every empty field is a question the model must answer, and "if not found, write 'not mentioned'" turns silent gaps into visible ones. This raised extraction recall by 15% and beat the other strategies tested, including self-reflection (customised prompting).
5. Force a second search. After the first list, "Wait, could there be other solutions?" resets focus; long-reasoning models lifted recall to 50% versus 15% for regular ones; merging 5-10 independent runs covers more still (Reasoning Overconfidence).
6. Go through every category explicitly. A format that requires a verdict and confidence for each label stops the model from dropping later categories; disagreements between independent labelers go to a short debate (CFD).
7. Track required vs confirmed vs missing. Structured Evidence Assessment keeps a Required Findings list and names Remaining Gaps before answering; Questions-of-Thoughts has the model ask itself "what if..." questions per module so edge cases are spelled out before the final text.

A combined prompt stack

1. Define coverage: name the categories or required fields up front, or have the model derive a rule.
2. Template: one field per item, with "not mentioned" or "no data in the text" as an allowed value.
3. Batch: for more than 20-30 checks, split into groups and require a quote before each verdict (sharding).
4. Second pass: ask what was missed, or run a skeptical critic over the list; for high stakes, add perspectives (LLM Council uses independent experts, anonymous critique and a chair who prioritizes what at least two flagged).
5. Verify the rule: run one known-good example through it before trusting the checklist.

What does not work

• "List ALL the options" in capitals. Emphasis does not change how the list is built; the model still stops at the first few (Reasoning Overconfidence).
• Raising temperature. It adds word-level variety, not new solution paths; recall and calibration stayed put.
• More budget in one pass. Extra tokens and reasoning steps inside a single checklist request barely moved accuracy; group size did (sharding).
• Trusting "that's everything". Stated confidence was far above actual recall, so the model's own sense of completeness is not evidence.
• Huge quotas. Forcing a minimum count pushes the search further but, set too high, invites irrelevant filler; prefer categories over raw numbers.

I need a {checklist / rubric / list of criteria / answer key} for grading {task}.

Step 1. Don't list every possible variant right away. Formulate a GENERAL RULE —
what markers tell you whether a candidate fits or not.

Step 2. Test this rule against a known-correct example: {known_correct_example}.
Does it pass? If not, rewrite the rule.

Step 3. Now apply this rule to each candidate one at a time and give a verdict
of "fits/doesn't fit" with an explanation:
{candidate_1}
{candidate_2}
{candidate_3}

Here is the full context/document:
{document_text}

Check ONLY these items (the rest will be checked separately, don't worry about them):
{criteria_group}

For each item in this group:
1. Provide an exact quote from the document confirming or refuting the condition
2. Issue a verdict: passed / failed / no data in the text

--- (repeat for each group of criteria) ---

Final request:
Collect the verdicts from all groups into one table. Separately highlight items marked "failed" and "no data."

Analyze the {document/text/brief} across four categories.
Go through each category separately and completely before moving on to the next.

CATEGORY P — PROJECT:
Who is involved? What organizational constraints exist (timeline, budget, team)?
What context matters for understanding the task?

CATEGORY E — ENVIRONMENT:
Which external systems, services, tools are involved or mentioned?
What regulatory, legal or operational constraints are there?

CATEGORY G — GOALS:
What business goals are behind the task? What counts as success?
What should the user or client ultimately get?

CATEGORY S — SYSTEM:
What specifically must it do (features)?
What non-functional requirements: speed, security, reliability, scale?

For each category: {format — list / table / paragraphs}.
At the end: point out overlaps and contradictions between categories.

{document/text}