TL;DR
Self-Reflection Critique is a technique where a second LLM call critiques the output of the first. The first pass labels the text and gives a rationale. The second pass reads the original text plus the first pass's rationale, checks it for typical errors, and either confirms the label or overturns it.
Even when the first LLM classifier posts good metrics on a test set, it can be systematically wrong on real data. In the study, the first pass scored F1 between 0.74 and 1.00, yet manual review showed that 8-54% of positive labels were false. Two main errors: meta-discussion (the model confuses discussing a criterion with applying it) and misinterpretation (it violates the task definition and ignores exceptions).
The second pass gets special instructions: check whether the first fell into these traps. If all rationales are wrong, overturn the label. If at least one rationale holds up, keep it. This approach improved F1 by 0.04-0.25, and lifted the worst codes from 0.52-0.55 to 0.69-0.79.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1 (first call): Classification + rationale
Input: text + criterion
Output: label (yes/no) + why it was applied
STEP 2 (second call): Critique the positives
Input: text + criterion + rationale from Step 1
Task: check the rationale for typical errors
Output: confirm or overturn the label
Important: Step 2 is applied only to positive labels from Step 1. That saves tokens and focuses the critique where most errors live.
Example
Task: You're analyzing 500 reviews of Instacart. You're looking for mentions of product quality problems (spoiled, not fresh, wrong items in the order). You need to separate real complaints from mentions of the topic in some other context.
Step 1 — First pass (generous labeling):
Read this grocery delivery review. Determine whether it mentions a product quality problem (spoiled, not fresh, not what was ordered).
Criterion: The user complains about the condition of the products received or that they don't match the order.
Answer:
1. Label: YES or NO
2. Rationale: a quote from the review + why you applied the criterion
Review: {review text}
Step 2 — Critique the positive labels:
You are an expert reviewer. You are checking whether the first analyst applied the criterion "product quality problem" correctly.
You have:
- The review
- The criterion
- The first analyst's rationale
Check for TWO typical errors:
ERROR A (Meta-discussion): The rationale relies on a phrase where the author DISCUSSES the possibility of a problem rather than STATING that it happened.
Examples:
- "Was worried the produce wouldn't be fresh, but it was fine" — NOT a complaint
- "Good that they check quality before shipping" — discussion of the process
- "The products were spoiled" — a complaint ✓
ERROR B (Misinterpretation): The rationale relies on information not covered by the criterion.
Example: a problem with packaging or delivery time is mentioned, but not with the products themselves.
SUFFICIENCY RULE: If there are several rationales and AT LEAST ONE is valid, the label is CORRECT. One mistake does not cancel a strong valid statement.
Give your analysis and decision:
Analysis: [explain why the label is correct or which error applies]
Decision: {"label_correct": true/false, "error": ["A" or "B" if any]}
---
Criterion: {definition}
Review: {text}
First analyst's rationale: {rationale from Step 1}
Result:
After the first pass you'll get ~50-70 reviews labeled "quality problem." Some of them are false positives: the person mentioned the topic but wasn't complaining. The second pass checks every rationale: is this really a complaint or just a mention? It'll overturn 20-40% of the false labels, leaving only real problems.
Why It Works
The LLM weakness: The model sees keywords ("product quality") and fires too broadly. It struggles to tell use of a criterion (a complaint) from mention of a criterion (a discussion). It's even worse with nuances in definitions: if it says "don't code when merely mentioned," the model often ignores that on the first pass.
The LLM strength: The model is good at analyzing someone else's reasoning. When it sees a finished rationale and an instruction like "check whether it fell into these two traps," it gets more precise. The second pass focuses on specific errors instead of solving the task from scratch.
How the method exploits this: The first pass is tuned for high recall — catch everything that looks relevant, even if it overshoots. The second pass works as a precision filter — weed out the dubious stuff. Instead of one hard instruction ("be precise but don't miss anything"), you get two simple modes: "find everything" → "check what you found."
Control levers:
Error types in the critique: The example has two (meta-discussion, misinterpretation). You can add your own task-specific ones: "Error C: confused cause and effect" or "Error D: only considered part of the definition." The more precisely you describe the error pattern, the better the second pass will catch it.
Sufficiency rule: Right now it's "one error doesn't cancel a valid one." You can tighten it to "all rationales must be valid" for conservative classification. Or relax it to "at least two rationales" for a middle ground.
First-pass mode: In the example the prompt is neutral. You can explicitly write in Step 1: "When in doubt, label positive. Better to over-include than to miss." That raises first-pass recall; the second pass will filter out the excess.
Positives only, or everything: The study only critiqued positive labels (saves tokens, that's where most errors are). If your first pass misses important cases, add a critique of the negative labels: "Check whether the first analyst missed an implicit mention of the criterion."
Prompt Template
Step 1: First pass (classification)
Read the text and determine whether this criterion applies: {criterion_definition}
{additional_instructions_if_needed}
Answer:
1. Label: YES or NO
2. Rationale: quote a fragment of the text and explain why you did/didn't apply the criterion
Text:
{text_to_analyze}
Step 2: Critique of positive labels
You are an expert reviewer. You are checking the decision of a first analyst who applied a criterion to a text.
You have:
- The criterion definition
- The text
- The first analyst's rationale (they assigned the label YES)
Check the rationale for typical errors:
ERROR A (Meta-discussion): The rationale relies on a phrase where the author DISCUSSES the criterion's topic rather than USING it as a statement.
Signs: "I'm afraid that...", "good that they check...", "it would be bad if...", "it usually happens that..."
NOT an error: a direct statement of fact ("there was X", "Y happened")
ERROR B (Misinterpretation): The rationale relies on information not covered by the criterion definition.
Pay special attention to exceptions in the definition: "Do not code when...", "Not covered by this criterion..."
SUFFICIENCY RULE: If the first analyst gave several rationales and AT LEAST ONE of them is valid, the label is CORRECT.
One error does NOT cancel a strong valid statement.
{task_specific_clarifications}
Give your analysis:
Analysis:
[Explain: the label is correct, or which error applies. If there are several rationales, evaluate each one]
Decision:
```json
{
"label_correct": true/false,
"error": ["A" or "B" if applicable, otherwise an empty list]
}
Do not write anything after the Decision block.
Criterion definition: {criterion_definition}
Text: {text_to_analyze}
First analyst's rationale: {rationale_from_step_1}
**What to fill in:**
- `{criterion_definition}` — a clear description of what you're labeling. With examples of what's in and what's excluded.
- `{text_to_analyze}` — the piece of text to analyze (a review, a message, a paragraph of a document)
- `{additional_instructions_if_needed}` — task specifics, edge cases
- `{rationale_from_step_1}` — the rationale from the first call (save it between steps)
- `{task_specific_clarifications}` — if you see recurring errors, add 1-2 specific instructions
**Workflow:**
1. Run Step 1 on the whole dataset
2. Select all texts where label = YES
3. For each one, run Step 2 (pass the text + the rationale from Step 1)
4. If Step 2 returns `"label_correct": false` — remove it from the positives
---
🚀 **Quick start** — paste into the chat:
Here is a two-pass verification scheme for text classification. Adapt it to my task: [describe what you're analyzing and which criterion you're looking for].
Ask me questions to fill in: 1. The exact criterion definition (with examples of what's in/out) 2. Typical classification errors for this criterion 3. Which edge cases matter in my task
[paste the template above]
The LLM will ask about the specifics of your criterion: what definitely belongs, what definitely doesn't, which phrases cause confusion. It'll take the pattern from the template and fill it in with the details of your task. After the clarifications you'll get two ready-made prompts: one for labeling, one for critique.
---
## Limitations
> ⚠️ **Workflow hurdle:** You need two sequential calls. In between, you have to save the rationale from the first and pass it along. In a regular chat that's manual (copy → paste into the next prompt). Automating it takes an API or a script.
> ⚠️ **Tokens double:** The second pass rereads the text + the first pass's rationale. For long texts or big datasets that gets expensive. To save: apply the critique only to positive labels (as in the study), not to all of them.
> ⚠️ **You need an error typology:** The method works when you know **which** errors are typical. If the task is new, first run the first pass on 50-100 examples, check by hand where it messed up, and write down the patterns. Without that, the second pass doesn't know what to look for.
> ⚠️ **Rare categories:** If the criterion appears in 1-2% of texts, the first pass will find few positives and there's almost nothing to critique. The method pays off when there's something to filter (at least 10-20 positive labels per hundred texts).
> ⚠️ **Not for subjective judgments:** This technique is about sticking to clear definitions. "Does the text meet criterion X?" — good. "How creative is the text?" — bad. The second pass adds no new information; it checks the logic of how the definition was applied.
---
## How They Studied It
The team analyzed 3,000 email messages from the **Apache Software Foundation** — project discussions ahead of graduation decisions. They developed 6 codes (evaluation criteria): community engagement, corporate support, fit with Apache culture, policy compliance, mentoring quality, technical potential.
**First** they trained an initial LLM classifier on 120 labeled emails. On the test set they got **F1 from 0.74 to 1.00** — looks great! They ran it on the whole corpus. Then they took a random sample of positive labels (60 emails per code) and checked them by hand. **Bummer: 8-54% of labels were false.** For two codes, more than half of the positives were junk.
**Why?** The test set didn't cover the full variety of errors. Manual review turned up two recurring patterns:
1. **Meta-discussion:** The email **discusses** the criterion (arguing whether it matters, whether it's sufficient) rather than **applying** it to evaluate the project. Example: "It's unclear whether license compliance is part of the graduation criteria" — the model counted this as a mention of "policy compliance," even though it's a debate about the rules, not an evaluation of the project.
2. **Misinterpretation:** The model ignored exceptions in the definition. Take the "mentoring" code: the definition says "don't code if a mentor is merely mentioned; code only when the quality of their involvement is being evaluated." The model blew that off and labeled any mention of a mentor.
**Next** they built a prompt for the second pass. It has two blocks of checks (one per error found) + the rule "if even one rationale is valid, keep the label." They ran the second pass on the same 60 emails per code. **Result: the second pass caught 75-84% of the false positives** (error recall). Precision varied: for one code it was too conservative (missed some errors), for another it sometimes overturned valid labels.
**Final test:** They took a fresh random sample of 150 emails from the whole corpus (untouched during development). They ran the two-pass pipeline and compared it with manual labels. **For the four problem codes, F1 rose by 0.04-0.25.** The two worst codes went from 0.52-0.55 to 0.69-0.79. The two codes that were good to begin with (false-positive rate <10%) barely changed — there was nothing to fix.
**Key insight:** Even when the first classifier posts decent metrics on a controlled test, on real data it can systematically slip up on specific patterns. Manual review of a sample of positives + an error typology + a targeted critique for those errors is an easy way to boost precision without retraining the model.
---
## Adaptations and Extrapolations
### 🔧 Technique: Critique the negative labels → catch the misses
The study only critiqued positive labels (that's where most errors are). But if your first pass is too conservative and **misses** important cases, add a third step: a critique of the negatives.
You are an expert reviewer. The first analyst assigned the label NO for the criterion: {criterion_definition}
Check: is there an IMPLICIT or INDIRECT mention of the criterion in the text?
Typical misses: - The criterion is expressed with synonyms or paraphrase - The criterion is implied by context - The author hints at the criterion without naming it directly
If you find a mention, flip the label to YES. If the NO label is justified, confirm it.
Text: {text}
This costs more (you're checking all the negatives too), but it raises recall.
---
### 🔧 Technique: Iterative critique → rounds of review
Instead of one round of critique, run several. Each critic reviews the previous one's decision.
Round 1: The first critic checks the first analyst's rationale Round 2: The second critic checks the first critic's decision "Was the first critic too strict? Did they overturn a valid label?" Round 3: The third critic looks at contradictions between Rounds 1 and 2 "If the opinions diverged, who's right? Why?"
Apply it only to borderline cases (where the label changes between rounds). Expensive, but for high-stakes tasks it boosts reliability.
---
### 🔧 Technique: Specialized critics for each error type
Instead of one all-purpose critic, use several specialized ones. Each one hunts for its own error pattern.
Critic A: Checks only for meta-discussion "Is this a statement of fact or a discussion of a possibility?"
Critic B: Checks only for fit with the definition "Is this covered by the definition? Are any exceptions violated?"
Critic C: Checks only edge cases "Is this a central example of the criterion or a peripheral one?" ```
Run them in parallel. Overturn the label only if all critics agree it's an error.
Resources
Self-reflection in Automated Qualitative Coding: Improving Text Annotation through Secondary LLM Critique
Zackary Okun Dunivin (University of Stuttgart), Mobina Noori (University of California Davis), Seth Frey (University of California Davis), Curtis Atkinson (University of Washington)
Related work: - Dunivin (2025): Adapting human codebooks for LLM interpretation - Törnberg (2024): LLM annotation as a methodological "wild west" - Ziems et al. (2024): Guidelines for validating LLM annotations - Madaan et al. (2023): Self-Refine — iterative improvement through feedback - Shinn et al. (2023): Reflexion — agent systems with reflection