Promptimizer: Structured Prompts for Personal Content Classification

INITIALIZATION (one prompt):

1. Describe the task in broad strokes
2. The LLM generates a draft: [description] + [positive rubrics] + [negative rubrics]
3. Label 20 "interesting" examples (where the LLM is unsure)
4. The LLM refines the rubrics based on your labels
→ Structured prompt is ready

ITERATION (several requests):

STEP 1: Find a classification error
STEP 2: Explain WHY it's an error (or pick from the LLM's options)
STEP 3: The LLM generates 2-4 targeted edits (only to rubrics, not the whole prompt)
STEP 4: Pick the best edit
→ Updated prompt with understandable changes

APPLICATION (5 requests per classification):

One comment/text → run it 5 times with a different order in the batch
→ Collect the votes → go with the majority
→ Consistent result + a confidence measure

Task: You're filtering comments on your startup-focused Discord server. You want to surface constructive criticism — not just "this sucks," but reasoned breakdowns with alternatives.

Prompt (initialization):

You are classifying comments in a community about startups.

GENERAL DESCRIPTION:
Comments with constructive criticism — where the person doesn't just bash,
but explains WHAT isn't working and HOW it could be done differently.

POSITIVE RUBRICS (catch):
- Points out specific problems with reasoning
- Proposes an alternative solution or approach
- Backs it up with experience, data or case studies

NEGATIVE RUBRICS (don't catch):
- Just "bad idea," "it'll never take off" with no explanation
- Generic lines like "use your head"
- Personal attacks instead of addressing the substance

EXAMPLES:
[4 labeled comments marked CATCH/DON'T CATCH]

Classify the comment: [text]
Answer: CATCH or DON'T CATCH

Iteration after an error:

The filter missed: "The idea is half-baked, but if you switch monetization to subscriptions instead of ads,
it'll land. Here's how Notion does it."

Why it's an error: There's a concrete proposal (subscriptions instead of ads) with an example.
This is constructive criticism, even if it's short. "Half-baked" is a judgment, but a solution follows.

The LLM will generate rubric edits:
→ Option 1: Add to positive: "A short, concrete proposal with a real-world example"
→ Option 2: Refine in positive: "Proposes an alternative → including short formats with case studies"

You pick Option 1 → rubric added → prompt updated.

Result: Each iteration adds or refines one rubric. You see what changed and why. After 3-5 edits the prompt accurately reflects your idea of "constructive criticism" — and you can explain every rule.

The LLM's weakness: Automatic prompt optimization produces unreadable results. The algorithms take random errors and let the LLM rewrite the prompt however it likes. What you get: long descriptions, duplication, every error baked in as an example. After 3-4 rounds the prompt becomes a wall of text — the user can't follow the logic and can't fix it.

The LLM's strength: The model is great at making targeted edits to structured text if you explain the problem. Instead of "come up with a new prompt" — "here's error X, here's why it happened, add a rule to account for it." The LLM generates a surgical edit without breaking the structure.

How the method exploits this: The prompt is split into blocks (description, positive rubrics, negative rubrics). Iterations are limited to four actions: add a positive rubric, add a negative rubric, edit an existing positive rubric, edit an existing negative rubric. Every edit is targeted — aimed at a specific error with an explanation. The structure is preserved and the changes are transparent.

Control knobs:

• Number of examples: 4 is optimal (more = less readable, negligible accuracy gain). You can add 5-6 for a hard task.
• Majority voting (5 requests): Expensive, but it gives you consistency. To save money, use it only on the final version of the filter. For a draft, one request is enough.
• Type of edit: If you see a negative rubric is too broad, refine it instead of adding a new one. Fewer rubrics = a clearer prompt.
• Explaining errors: The more precisely you describe WHY it's an error, the more targeted the edit. "This isn't constructive" → a vague edit. "There's a solution + a case study here, but no explanation of the problem — this is the borderline" → a sharp edit.

You are classifying {content type: comments/emails/posts}.

GENERAL DESCRIPTION:
{The task in broad strokes — what to catch and why}

POSITIVE RUBRICS (catch):
- {Specific signal 1}
- {Specific signal 2}
- {Specific signal 3}

NEGATIVE RUBRICS (don't catch):
- {Anti-signal 1 — looks similar but isn't it}
- {Anti-signal 2}
- {Anti-signal 3}

EXAMPLES:
1. Text: "{example 1}" → CATCH, because {reason}
2. Text: "{example 2}" → DON'T CATCH, because {reason}
3. Text: "{example 3}" → CATCH, because {reason}
4. Text: "{example 4}" → DON'T CATCH, because {reason}

Classify:
Text: "{text to classify}"

Answer: CATCH or DON'T CATCH
Explanation: {brief reasoning}

Fill in:

• {content type} — what you're filtering (Discord comments, customer emails, social media posts)
• {task} — what to catch (toxic comments, high-ticket leads, audience insights)
• {specific signal} — objective criteria (mentions numbers, contains a question, cites a case study)
• {anti-signal} — similar but not it (a plain complaint vs. constructive criticism)
• {examples} — 4 real examples with an explanation of the decision

After the first run:

1. Find a classification error
2. Explain why it's an error: "Text X was classified as DON'T CATCH, but it contains {signal}. This is {a borderline case / a clear hit}, because {reason}."
3. Ask for an edit: "Extend the POSITIVE RUBRICS to catch cases like this" or "Refine the rubric '{name}' to exclude false positives"
4. The LLM will offer 2-4 options — pick the best
5. Update the prompt with the new rubric

For consistency (optional, expensive):

Run this prompt 5 times on the same text.
Count the votes: how many times "CATCH" vs "DON'T CATCH".
Go with the majority decision.

🚀 Quick start — paste into the chat:

Here is a template for building a personal content filter. Adapt it to my task: {your task — what you're filtering and why}.

First ask me:
- What content am I filtering (comments, emails, posts)?
- What do I want to catch, in broad strokes?
- Do I have examples of things that definitely should and definitely shouldn't be caught?

Then generate a draft prompt using this structure:
[paste the template above]

After that, give me 10-15 borderline examples to label — cases where it's unclear whether to catch them or not. Refine the rubrics based on my labels.

The LLM will ask about the content type, the filter's goal and borderline cases — so it can build relevant rubrics and examples. It will take the structure from the template (description + positive/negative rubrics + 4 examples) and plug in your data.

⚠️ Cost: Majority voting takes 5 requests per classification. For 100 comments = 500 requests. Use it on the final version of the filter, not the draft. For a quick check, one request is enough.

⚠️ Requires discipline: The method demands a methodical approach — find the error, explain it, request an edit, pick an option. It's not a "magic button" but an iterative process. 3-5 edits are usually enough for a stable filter.

⚠️ Subjective tasks: It works for tasks with relatively clear criteria ("constructive criticism," "mentions a price," "question about a feature"). For highly subjective ones ("interesting post," "creative idea") the criteria are fuzzy and rubrics help only a little.

⚠️ Not for simple filters: If a couple of keywords solve the task ("mentions Python"), the structure is overkill. The method is for complex filters where the line is blurry ("constructive vs. just ranting," "lead vs. spam").

The team built a system called Puffin for YouTube creators — filtering comments on videos. They compared two approaches: Promptimizer (human-in-the-loop) vs. ProTeGi (automatic optimization).

Lab experiment (16 people): Participants built filters for two topics — political comments (moderation) and cultural experiences with food (curation). They first labeled 20 examples, then did 3 iterations with each method. Results were measured on a separate set of 100 comments that the participants themselves labeled as ground truth.

The finding: Accuracy (accuracy, F1) was comparable for both methods — 63-64% on test data. BUT all 16 participants unanimously preferred Promptimizer for iterating. Why? The automatic method made opaque edits — participants only saw the final prompt and didn't understand the logic of the changes. In Promptimizer participants chose which error to fix (precision vs. recall), saw why an edit was proposed, and understood what changed in the rubrics.

Field study (10 YouTube creators, 3 weeks): Channel owners with 500 to 1 million subscribers used Puffin on real comments. They built filters like "audience insights," "constructive criticism," "toxic attacks." Success: 6 out of 10 kept using it after the study.

Key insight: Users evolve in how they understand the task. At first — a vague idea ("I want to see useful comments"). After labeling examples — a sharper understanding ("useful = a specific question or a detailed personal experience"). After errors in real use — pinpoint refinements ("a useful experience, but not one that mentions other channels — that's self-

promotion"). Automatic optimization ignores this evolution — it just blindly minimizes errors on the initial labels. Promptimizer gives control — the user decides which errors matter more and how to fix them.

Promptimizer: User-Led Prompt Optimization for Personal Content Classification — Leijie Wang, Kathryn Yurechko, Amy X. Zhang (University of Washington, University of Oxford). Published at CHI'25.