STEP 1: Structured check against a checklist
→ List of potential problems (hints)

STEP 2: LLM evaluation with injected hints
→ In-depth analysis + new findings

You can do it in one prompt (ask the LLM to do both steps) or in two requests (you do the checklist, then feed it to the LLM).

Task: Review the text of a sales proposal for a client — a B2B warehouse inventory automation service

Prompt for a two-stage review:

Evaluate this sales proposal for a client. First, run a quick check against a checklist of typical B2B copy problems:

CHECKLIST (mark what you found):
- The client's benefit isn't in the first paragraph
- No concrete impact numbers (% savings, hours, dollars)
- ROI isn't calculated or is buried at the end
- Technical terms with no explanation
- No case study from a similar client
- Vague call to action ("get in touch with us")
- Risks/objections aren't addressed
- Pricing isn't transparent
- No urgency/decision deadline

After the checklist, do an in-depth evaluation:
1. Analyze each problem you found in detail
2. Find what the checklist doesn't cover
3. Suggest specific edits

PROPOSAL TEXT:
[your text]

Result: The model will show the filled-in checklist with marks (these are the hints), then an in-depth analysis of each item with examples from the text and specific recommendations. Plus it will find problems outside the checklist — a tone that doesn't fit the audience, weak wording, logical gaps.

The weakness: With a generic "rate the quality" request, an LLM doesn't know where to look — it does a surface check and misses omissions (what should be there but isn't), links between distant parts and implicit logic violations. The model sees what's written well and what's missing poorly.

The strength: An LLM is great at directed analysis — give it concrete leads ("check the variable initialization", "is there an ROI calculation") and it will check exactly that, thoroughly. Structured hints focus the model's attention on the critical aspects.

The mechanics: The checklist acts as external memory and a guide. You (or a simple script, or the LLM itself on a first pass) do a quick structured check — it's cheap and covers known patterns. You feed the findings into the LLM prompt as hints — the model goes deeper on each lead + uses its own strengths for unexpected finds. A hybrid: structure finds the obvious, the LLM finds the non-obvious.

Prompt control levers:

• Checklist detail: 5 items → fast but narrow; 30 items → broad but slow. For simple tasks, trim the checklist down to the critical items.

• Hint format: you can pass them as a list ("Found: X, Y, Z") or as structured XML (<hint type="missing">no ROI</hint>). XML is sharper for complex checks.

• Instruction after the hints: add "be sure to check every hint in detail" if the model is superficial, or drop it if the hints are enough on their own.

• One-pass vs Two-pass: you can ask the LLM to do both steps in one prompt (fewer requests) or split it into two (you control the checklist manually). Two-pass for critical tasks, one-pass for speed.

Option 1: Single request (the LLM does both steps)

Evaluate {object of evaluation} in two stages:

STAGE 1 — Quick check against the checklist:
{checklist of typical problems for your task}

Mark what you found from the checklist.

STAGE 2 — In-depth evaluation with hints:
1. Analyze each problem found from the checklist in detail
2. Find problems the checklist doesn't cover
3. Give specific recommendations

OBJECT OF EVALUATION:
{text/code/idea to evaluate}

What to plug in: - {object of evaluation} — what we're checking (text, code, business plan, design mockup) - {checklist} — a list of 5-15 typical problems for your domain (see examples below) - {text/code/idea} — the specific material to check

Option 2: Two requests (you build the checklist)

Request 1 (building the checklist):

Create a checklist of 10-15 typical problems for evaluating {content type}.

Checklist requirements:
- Concrete, checkable items (not "poor quality" but "no impact numbers")
- Cover the critical aspects: structure, logic, completeness, style
- Quick to check (yes/no, present/absent)

Context: {description of the task and audience}

Request 2 (evaluation with hints):

Evaluate {object of evaluation}.

HINTS — the preliminary check found:
{list of problems you found with the checklist}

Tasks:
1. Analyze each problem from the hints in detail — show examples from the text
2. Find problems not covered by the hints
3. Suggest specific edits

OBJECT OF EVALUATION:
{text/code/idea to evaluate}

What to plug in: - {content type} — social media posts, technical articles, startup pitches, Python code - {description of the task} — who the reader is, what's critical, which standards apply - {list of problems} — conclusions from the checklist pass (3-7 items)

Sample checklists for different tasks

For sales copy (proposals, landing pages): - The client's benefit isn't in the first paragraph - No concrete numbers (%, $, hours) - No social proof (case studies, testimonials) - Technical terms with no explanation - Vague call to action - Objections aren't handled

For technical articles/documentation: - No usage example in the first 2 paragraphs - Terms introduced without definitions - Instruction steps aren't numbered - No visuals (diagrams, code) - Edge cases / limitations are missing - No "quick start" for beginners

For business plans/strategies: - Goals aren't measurable (no metrics) - No unit economics calculation - Competitors aren't analyzed - Risks aren't named - No plan for failure (Plan B) - Unrealistic timeline

For code (code review): - Functions >50 lines (not decomposed) - No error handling (try/catch) - Magic numbers (hardcoded values) - Unclear variable names (a, tmp, data) - No comments on complex logic - Duplicated code (copy-paste)

🚀 Quick start — paste into the chat:

Here is the Analytic Hints template. Adapt it to my task: {your evaluation task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what we're evaluating, who the audience is and which aspects are critical — so it can put together a relevant checklist. It will take the two-stage evaluation pattern and adapt it to your domain.

⚠️ Depends on the checklist: If the checklist is incomplete, the hints won't cover every problem. The model will find some things on its own, but the method's main power lies in the quality of the checklist. You need to update the checklist as you discover new error patterns.

⚠️ Narrowed attention: Hints improve checking against the checklist, but they can narrow the evaluation — the model looks less for problems outside the hints. In the study, models with hints found 94% of the checklist errors but only 67% of their own original findings (100% without hints). The balance: hints for known patterns, but don't forget "find what else" in the prompt.

⚠️ Subjective criteria: The method is strong for checkable aspects (element present/absent, logical sequence, completeness). For subjective judgments ("how creative", "emotional impression") hints add less.

A team from IBM Research took a production task — evaluating generated COBOL code (the language of legacy systems at banks and government agencies). They used four production-level LLM judges (Llama-4-Maverick, Llama-3-405b, DeepSeek-v3, GPT-OSS-120B) that were already running in their automated code evaluation pipeline.

They created 100 programs with deliberately planted bugs — realistic domain-specific bugs that experts confirmed as typical. Run through the LLM judges without hints, each one found only 45-53% of the errors. Experts manually analyzed the missed bugs and built a taxonomy of blind spots — six categories of typical LLM misses.

Based on the taxonomy, they wrote a lightweight analytic checker (pattern matching) for 30+ error types specific to COBOL. The checker produced short structured hints like "missing status check after CALL" or "file used before OPEN". The hints were injected into the judge's prompt in two ways: naive (simply appended to the end of the prompt) and optimized (explicitly asking the model to check each hint).

The results were dramatic: the hybrid setup (checker + LLM) raised coverage to 63-94% depending on the model. The best result — DeepSeek-v3 and GPT-OSS-120B with optimized hints: 94.4% coverage versus 45% for the original. A 2x jump.

A curious detail: they checked whether the model kept its original findings (the ones it made without hints). It turned out hints shifted the focus slightly — the models got better at catching checklist errors but partly lost their unique findings. DeepSeek-v3 kept 70% of its original findings, Llama-4-Maverick only 45%. That's an important insight: hints direct attention but can narrow the view — the model "wanders" through the code less and focuses more on the hints.

Why this matters in practice: The study showed that even production-level judges with tuned prompts and human-in-the-loop testing have systematic blind spots. It's not a bug in one model — it's a feature of the architecture. LLMs struggle to see omissions (what should be there but isn't), implicit links between distant pieces of code and violations of the order of operations. Analytic hints make up for these weaknesses with structure, no retraining required.

🔧 Technique: Self-generated hints for any task

If you don't have a ready-made checklist or analytic checker, ask the LLM to create hints for itself:

Task: Evaluate {object}

STEP 1: Create a checklist of typical problems for this type of content.
Format: a list of 10-15 items, each one checkable (present/absent).

STEP 2: Quickly check {object} against the checklist. Write down what you found.

STEP 3: Using the findings from Step 2 as hints, do an in-depth evaluation:
- Analyze each problem in detail
- Find problems outside the checklist
- Give specific edits

OBJECT:
{your text/code/idea}

Why: The model creates hints on the fly. No expert taxonomy needed — the LLM knows the typical patterns for most tasks.

🔧 Technique: Hints from competing agents

Combining Analytic Hints with Multi-agent debate:

Evaluate {object} using competing roles:

AGENT 1 — Critic (looks for problems):
Quickly check {object} against the checklist:
{your checklist}
Write down every problem found as hints.

AGENT 2 — Defender (justifies the decisions):
Read the Critic's hints. For each one:
- Is it a real problem or a false positive?
- If real, how critical is it?

AGENT 3 — Judge (final evaluation):
Taking into account the Critic and Defender's debate:
1. Confirmed problems and their priority
2. Problems outside the checklist
3. Specific recommendations

OBJECT:
{text/code/idea}

Effect: The Critic's hints pass through the Defender's filter — fewer false positives. The Judge sees arguments from both sides — a deeper final evaluation.

🔧 Technique: Cumulative hints for a series of reviews

If you're evaluating several similar objects (say, 10 blog posts), accumulate hints between reviews:

Review of text #1:

Evaluate the text against the checklist:
{standard checklist}

After the evaluation: Did you find new types of problems not covered by the checklist? Add them to the checklist.

Review of text #2:

Evaluate the text against the updated checklist:
{checklist + new items from review #1}

After the evaluation: Did you find any more new problems? Add them to the checklist.

Effect: The checklist evolves — it becomes specific to your task. By the 5th-6th review it covers 90%+ of the typical problems in your content.

🔧 Technique: Hints as prompt debugging

Use Analytic Hints to debug your own prompts:

Here is my LLM prompt:
{your prompt}

STEP 1: Check the prompt against the checklist:
- Is there a role/context?
- Is the task stated unambiguously?
- Are there examples (few-shot)?
- Is the output format specified?
- Are there constraints (what NOT to do)?
- Is the prompt not overloaded (< 1000 words)?

STEP 2: For each problem found:
- Why is it a problem?
- A specific fix

STEP 3: The rewritten prompt

Effect: A structured review of the prompt, treated like code — you find weak spots that stay invisible when you "just read it".

Paper: "Beyond Blind Spots: Analytic Hints for Mitigating LLM-Based Evaluation Pitfalls"

Authors: Ora Nova Fandina, Eitan Farchi, Shmulik Froimovich, Raviv Gal, Wesam Ibraheem, Rami Katan, Alice Podolsky

Organization: IBM Research, Israel

Dataset and prompts: https://github.com/AlicePod1/analytic-hints-paper

Related techniques from the paper: - Chain-of-Thought — the basic LLM reasoning technique - Self-Refine — iterative improvement through self-feedback - Progressive-Hint Prompting — reusing previous answers as hints - Directional Stimulus Prompting — generating hints via a policy model