TL;DR
Researchers compared how 5 prompt types perform with 6 models on the task of screening scientific papers. Zero-shot (no examples) maximizes recall: it finds almost everything relevant, but drags in a pile of junk. CoT-few-shot (reasoning + examples) gives the best balance: fewer false positives while keeping good coverage. Self-reflection (the model checks itself) performs worst of all: it includes way too much extra stuff while trying to justify itself.
The main pain point: when you ask an LLM to "find all matching options", it either misses important items (low recall) or hauls in everything in sight (low precision). Simple prompts (zero-shot) cast a wide but sloppy net. Structured prompts (CoT-few-shot) filter more accurately, but may let something slip. A model with self-reflection starts rationalizing its mistakes instead of fixing them ("this is relevant because you could read it this way") and drags in even more junk.
The fix: a staged approach, two passes with different prompts. The first pass is zero-shot for a wide net (miss nothing). The second pass is CoT-few-shot only on the borderline cases (cut the junk, keep the hits). That way you keep the coverage of the first pass and get the precision of the second.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STAGE 1 (Wide Net): Zero-shot prompt
→ Maximum recall, lots of candidates
→ Goal: don't miss anything important
STAGE 2 (Precision Filter): CoT-few-shot prompt
→ Only on borderline cases from stage 1
→ Goal: cut false positives
→ Keep the true matches
Result: High recall + High precisionBoth stages are separate requests in a regular chat.
Example in Practice
The task: You're hunting for the best articles on AI in US retail for an investor presentation. You need to (1) not miss important case studies and (2) not drag in every random piece about "digital transformation in retail".
Stage 1 prompt (Wide Net, Zero-shot):
From this list of articles, select everything that is in any way related to the use of artificial intelligence in US retail:
[a list of 50 articles with titles and abstracts]
Criterion: the article mentions AI/ML/neural networks AND the US retail market.
Answer with a list of the numbers of relevant articles.Stage 2 prompt (Precision Filter, CoT-few-shot on borderline cases):
Several of the selected articles are borderline cases. For each one, explain step by step:
1. Which specific AI technology is mentioned?
2. Is this a deployed case study or just a mention of a trend?
3. Are there concrete outcome metrics?
Examples:
- "Walmart deployed computer vision for shelf monitoring, accuracy rose by 23%" → RELEVANT (concrete case + metrics)
- "An expert believes it's time for retail to adopt AI" → NOT RELEVANT (opinion without a case)
Borderline articles:
[articles from stage 1 where the model was unsure]
For each: reasoning → verdict (RELEVANT/NOT RELEVANT).Result:
The first pass will return 20-25 articles out of 50: a wide net that may include pieces "about digital in general". The second pass will weed out the ones without specifics and leave 8-12 relevant case studies with metrics and rollout details. You get both coverage (nothing important missed) and precision (no junk in the final list).
Why It Works
LLMs struggle to hold two conflicting criteria at once: "find EVERYTHING relevant" (recall) vs "don't drag in junk" (precision). In a single prompt, the model picks one. Zero-shot without examples leans toward over-inclusion: the model reads "in any way related" as broadly as possible, because it doesn't know where the boundaries are. CoT-few-shot with examples leans toward caution: the model sees concrete "this is a yes, this is a no" samples and filters more strictly.
An LLM's strong suit is following a pattern from its context. Few-shot examples ("here's a good case, here's a bad one") calibrate the model: it starts demanding specifics instead of abstractions. CoT (chain of thought) makes the criteria explicit: "are there metrics? is the technology named?" instead of a vague "looks like AI in retail".
The staged approach uses both sides: first, the maximum sensitivity of zero-shot captures every candidate, then the strict CoT-few-shot filter cuts the false positives. You don't have to choose between recall and precision; you get both.
Control knobs:
- Width of the first pass: Make the criterion even looser ("mentions anything at all about AI") → you'll catch even more, but the second pass will take longer
- Strictness of the examples in the second: Show very concrete cases → the model gets pickier
- Number of passes: Add a third pass for the "really borderline" ones → even finer tuning
Self-reflection flopped because the model starts rationalizing its mistakes: "Hmm, this article is about digital, and AI is digital, so it's relevant", and drags in junk complete with a "justification".
Prompt Template
Stage 1: Wide Net (Zero-shot for maximum recall)
From the list of {item_type}, select everything that is in any way related to {your_task}.
{item_list}
Inclusion criterion: {broad_criterion}.
Answer with a list of the numbers/names of relevant items.Stage 2: Precision Filter (CoT-few-shot for precision)
Some of the selected items are borderline cases. For each one, explain step by step:
1. {check_criterion_1}
2. {check_criterion_2}
3. {check_criterion_3}
Worked examples:
- "{relevant_example}" → RELEVANT ({why})
- "{irrelevant_example}" → NOT RELEVANT ({why})
Borderline items:
{items_from_stage_1}
For each: reasoning → verdict.Notes:
{broad_criterion}is a loose condition that captures as much as possible ("mentions topic X"){check_criterion_N}are concrete questions for the second pass ("Are there numbers? Is the technology named?"){examples}are real samples from your field; show the boundary between "yes" and "no"
Limitations
⚠️ Two requests instead of one: The staged approach requires manually passing results between stages, copying the candidates from the first prompt into the second. With hundreds of items, that gets clunky.
⚠️ Subjective criteria: If "relevance" is fuzzy ("creative idea" vs "boring idea"), even CoT-few-shot won't give stable results; the model will still be guessing.
⚠️ Class imbalance: If relevant items are very rare (5 out of 1000), even the staged approach can miss the rare cases; zero-shot isn't all-powerful either.
How They Studied It
The team took 1,376 scientific papers on automating systematic literature reviews and hand-labeled them at two levels: (1) whether the paper is relevant to SLR automation (yes/no), and (2) which SLR stages it covers (search, screening, extraction, synthesis, writing) + whether it uses LLMs. That produced a gold standard: expert labels to compare against.
Then they ran all 1,376 papers through 6 models (GPT-4o, GPT-4o-mini, DeepSeek, Gemini-2.5-Flash, Claude-3.5-Haiku, Llama-4-Maverick) with 5 prompt types (zero-shot, few-shot, CoT, CoT-few-shot, self-reflection). That's 30 model+prompt combinations in total. For each one they computed accuracy, precision, recall, F1 against the expert labels.
Key finding: CoT-few-shot delivered the best F1 (0.913), balancing precision and recall. Zero-shot delivered the highest recall (0.971): it found almost everything, but with a pile of junk (precision 0.834). Self-reflection flopped: lowest F1 (0.835), worst precision (0.753), because the model tried to justify its mistakes instead of fixing them.
Why the staged approach follows from this data: If zero-shot finds 97% of what's relevant (but with 16% junk), and CoT-few-shot finds 93% of what's relevant (but with 10% junk), then combining them gives you the best of both: a wide net on the first pass → precise filtering on the second. This wasn't directly tested in the paper, but it follows logically from the precision-recall trade-off.
Interesting bit about the models: GPT-4o-mini turned out to be very cost-effective: with CoT-few-shot it hit F1=0.910 at $0.18 per 1,000 papers, while GPT-4o cost $3.04 for nearly the same F1=0.918. DeepSeek also struck a good balance: F1=0.914 at low cost.
On self-reflection: This was the only prompt that made results worse for almost every model. The mechanism: the model generates an answer → critiques it → revises it. But on classification tasks it starts rationalizing its original decision ("Well, actually, you could argue this paper is also about automation...") instead of checking honestly. It's a classic problem with self-reflection that lacks external grounding: the model talks itself into things.
Adaptations and Extrapolations
💡 Adaptation for hiring: Resume screening: use a wide pass to pick everyone with the right skills, then a second pass to cut those whose skills exist only in their job title.
Stage 1:
From these 100 resumes, select everyone who has worked with Python and data in any way.
Criterion: the resume mentions Python AND data (analysis/processing/visualization).Stage 2:
For each shortlisted candidate, analyze:
1. Which specific Python libraries do they name? (pandas, numpy, sklearn...)
2. Do they describe data projects, or just list it under skills?
3. Are there metrics for their results? (sped up processing by X%, built a model with Y% accuracy)
Examples:
- "Built a data processing pipeline in pandas, cut reporting time by 40%" → FIT
- "Skills: Python, working with data, Excel" → NOT A FIT (no specifics)
[candidates from stage 1]🔧 Technique: Add temperature to the stages → control strictness
For the wide net, use temperature=1.0 (more variability, fewer misses). For the precision filter, use temperature=0 (determinism, strict scoring against the criteria).
💡 Extrapolation: Staged prompting for review analysis
First pass: zero-shot to find every mention of product problems (wide). Second pass: CoT-few-shot to cut plain grumbling and keep bugs and concrete defects.
Stage 1:
From these reviews, select all the ones where the customer complains about problems with {product}.
[500 reviews]
Stage 2:
For each complaint, explain:
1. Is this a concrete product defect or general dissatisfaction?
2. Are the steps to reproduce the problem described?
3. Does it say what exactly doesn't work?
Examples:
- "The app crashes when I tap the Pay button on iOS 17" → BUG (specific, reproducible)
- "Awkward to use, everything is badly made" → NOT A BUG (emotion without details)Resources
Evaluating Prompting Strategies and Large Language Models in Systematic Literature Review Screening: Relevance and Task-Stage Classification by Binglan Han, Anuradha Mathrani, Teo Susnjak (Massey University, New Zealand)