Task: You're pulling customer feedback on a product that contains specific improvement suggestions (not just "it's bad," but "add feature X" or "fix the bug with Y"). You need to filter hundreds of comments automatically.

Iteration 1 — A bad prompt:

Determine whether this review contains a constructive suggestion.

Text: [customer review]

Answer: Yes or No

Iteration 2 — Sharpened the definition:

A constructive suggestion is when the customer describes:
- A specific feature that should be added
- A specific bug or problem that should be fixed
- A specific change to the interface or how the product works

Does NOT count as a suggestion:
- General praise or criticism without details
- An emotional reaction without saying what to change

Does this review contain a constructive suggestion?

Text: [customer review]

Answer: Yes or No

Iteration 3 — Asked the model to rephrase the best version:

Here is my classification task. Generate 10 alternative wordings of the definition of a "constructive suggestion", varying the structure, emphasis and examples.

[paste the prompt from Iteration 2]

Result: The model gives you 10 variants. You test each one on your 20-30 reviews with known answers (that you labeled yourself), count the match rate and pick the best. The gap between the worst and best variant can be 30-50% accuracy on your data.

An LLM doesn't understand the task "in general" — it looks for patterns in your words. If you write "constructive suggestion" without a definition, the model pulls the meaning from its training data — and there it could mean anything. If you define it precisely ("a specific feature" + "a specific bug"), the model knows what to look for.

Popular techniques work by amplifying the base prompt. A persona ("you are a product manager") or CoT ("think step by step") tries to activate extra context from the training data. But if the base definition is blurry, the extra context is blurry too. It's like trying to sharpen a blurry photo — doesn't work. First you need a sharp photo (the definition), then you can polish the details.

You can't predict the best wording because we don't know exactly which patterns the model latched onto in its training data. The word "constructive" can activate one set of patterns, "actionable feedback" another. The only way to find out is to test on real examples.

Control knobs: - Definition structure (list of criteria vs description vs examples) → vary it and test what works - Negative examples ("Does NOT count") → sharpen the boundaries, especially for fuzzy concepts - Few-shot examples (0 vs 3-5) → add them to the final version for +5-10% accuracy - Number of test iterations (5 vs 50 variants) → more iterations = better odds of finding the best prompt

Step 1: Define the construct as concretely as possible

Instead of abstract words, give an operational definition — what specifically has to be present in the text.

❌ Bad: "toxic comment"
✅ Good: "the comment contains: an insult aimed at a specific person OR a call to violence OR discrimination based on a trait (gender, ethnicity, religion)"

Step 2: Build a test set

Label 20-50 examples by hand. This is your benchmark for comparing prompts. Without it you can't tell which prompt is better.

Step 3: Write 5-10 prompt variants by hand

Vary: - The order of the criteria - The level of detail - Adding/removing negative examples - The format (list vs prose vs table)

Step 4: Test every variant

Run each prompt through your test set. Count the match rate against your labels.

Testing prompt:

Here is my classification prompt:
[your prompt]

Apply it to these texts and output the results as CSV:

Text 1: [text]
Text 2: [text]
...

Response format:
number,result
1,Yes
2,No
...

Step 5: Auto-generate variants of the best prompt

Take the best version from Step 4:

You are helping optimize a text-classification prompt.

Here is the current prompt:
[best version]

It scored 75% accuracy on the test data.

Generate 10 alternative wordings:
- Keep the meaning and the criteria
- Vary the structure, order, examples
- Try different ways of explaining it
- You may add/remove details

Output each variant as a separate block, numbered.

Step 6: Test all the new variants again

Repeat the Step 4 process with the new prompts.

Step 7: Add few-shot examples to the final version

Take the winner and add 3-5 examples with correct answers:

[your best prompt]

Examples:

Text: "Terrible product, everything is bad!"
Answer: No

Text: "Please add an Export to Excel button — right now I have to copy everything by hand"
Answer: Yes

Text: "Bug: clicking 'Save' throws a 404 error"
Answer: Yes

Now classify this text:
[new text]

⚠️ You need a labeled dataset: Without 20-50 examples with correct answers you can't compare prompts. You'll have to label them by hand.

⚠️ Iteration takes effort: To find the best prompt you need to test dozens of variants. That takes time, though it's a one-time investment.

⚠️ Doesn't work for very rare patterns: If the construct shows up in 2-3% of cases (positive meaning making in the study was at 17%), the model tends to answer "no" almost every time. You need special balancing techniques.

⚠️ Subjective constructs are harder: The more interpretive the task (not "is there a number" but "is there a deep psychological transformation"), the harder it is for the model and the more the precise definition matters.

The team took three classification tasks, from easy to hard. Easy: find gratitude in Reddit posts (an emotion people recognize easily — 75% agreement between annotators). Medium: find negative core beliefs in texts written by people with mental health struggles ("I'm worthless," "the world is dangerous" — 82% agreement). Hard: find positive meaning making — when a person describes growth through trauma ("after my mom died I became stronger" — 70% agreement).

On each task they tested thousands of prompt combinations: base definitions (handwritten and model-generated), personas ("you are a psychologist"), chain-of-thought ("think step by step"), explanations inside few-shot examples, zero-shot vs few-shot. Everything was compared against labels from expert psychologists.

The results were surprising: popular techniques like CoT and personas gave at most a 2-5% boost, and only when the base prompt was bad. The best result came from a combination: write the definition by hand → test variants → take the best → ask the model to generate more variants → test again. The gap between the worst and best prompt reached 75% accuracy on the hard task and 16% on the easy one.

Why: the model looks for patterns in the words of the prompt. Fuzzy words, fuzzy patterns. "Let's think" doesn't make up for "I don't know what to look for." Interestingly, auto-generated prompts sometimes beat human ones — apparently the model stumbled onto wordings that resonated better with its training data. But the most reliable approach is to combine: the human supplies the meaning, the model generates variations, and empirical testing picks the winner.

🔧 Technique: A/B-test definitions instead of a full sweep

If you don't have time for dozens of iterations, run a mini-test:

Here are two wordings of the same classification task.

Version A:
[first definition]

Version B:
[second definition]

Apply both to these 10 texts. For each text, output the result in this format:

Text 1:
- Version A: Yes/No
- Version B: Yes/No

[10 texts to classify]

Compare the results with your labels and pick the better version. It's not optimal, but it beats not testing at all.

🔧 Technique: Ask the model to explain why it got it wrong

Once you've found the best prompt but accuracy still isn't 100%, take the misclassified cases:

Here is my classification prompt:
[prompt]

Here is a text you classified as "Yes", but the correct answer is "No":
[the text you got wrong]

Explain:
1. Why did you decide it was "Yes"?
2. Which parts of the text misled you?
3. How should I change the definition in the prompt to avoid errors like this?

Use the model's explanations to refine the definition — add negative examples, clarify the boundaries.

🔧 Technique: Chain-of-thought for debugging, not for production

In the study CoT didn't improve accuracy. But it's useful for understanding why the model classifies things the way it does:

[your prompt]

Before answering, explain your reasoning step by step:
1. What key elements do you see in the text?
2. Do they match the criteria in the definition?
3. What doubts do you have?

Then give your final answer: Yes or No

Use this while developing the prompt to understand the model's logic. Drop the CoT from the final version — it adds tokens without improving accuracy.

Improving Alignment Between Human and Machine Codes: An Empirical Assessment of Prompt Engineering for Construct Identification in Psychology

Authors: Kylie L. Anglin, Stephanie Milan, Brittney Hernandez, Claudia Ventura
University of Connecticut (Department of Educational Psychology, Department of Psychological Sciences)

Works referenced in the study: - Automatic Prompt Engineering (APE) method, Zhou et al. 2022 - Chain-of-Thought papers: Wei et al. 2022 (few-shot CoT), Kojima et al. 2022 (zero-shot CoT) - Explanatory prompting: Lampinen et al. 2022 - GoEmotions dataset: Demszky et al. 2020