Meta-prompting: using the model to diagnose and improve your prompts

What meta-prompting is

Meta-prompting means pointing the model at the prompt instead of the task: diagnosing a draft, asking what it leaves out, expanding a one-line request into a spec, or distilling rules from examples and failures into a reusable instruction. It ranges from a single "critique my request before you run it" step to a loop that grows a playbook over dozens of attempts.

The studies in this hub agree on one thing: the model is a useful prompt editor only when it has something to measure against. A bare "improve my prompt" gives it no reference point, so it guesses. The PEEM evaluation study replaces that request with nine named axes, 3 for the prompt and 6 for the response, each scored 1 to 5 with a written rationale; the rationales become the edit list.

• Diagnose: score the prompt and its output on named criteria, not on overall quality.
• Expand: have the model turn a short request into a full brief or into clarifying questions.
• Distil: extract rules from good/bad example pairs or from logged failures.
• Regress: re-run every saved test case after each edit, so a fix does not silently break something else.

Why structured diagnosis beats "make it better"

Two mechanisms recur. First, a model asked to improve something vague fills the gaps with average defaults; asked to find specific defects, it names them. The Prompt Analyzer study on software feature requests had the model hunt for ambiguous phrases and missing data and return them as clarifying questions rather than invented details, and a few examples of "defects" in the prompt sharpened the analysis. The 5W3H expansion study found that an AI expansion of a one-sentence task across eight dimensions was statistically indistinguishable from a brief an expert wrote by hand.

Second, models extract rules well from contrast and failure, and badly from adjectives like "livelier" or "more professional". In Spec Learning, 20 good/bad answer pairs became a system prompt in two requests, and that prompt beat DPO fine-tuning on six of seven datasets with 50x less data. Two details from that work are worth copying: randomly sampled pairs beat hand-picked "most telling" ones, and a coherent second-person paragraph beat a numbered list of the same rules. The playbook-induction study shows the same pattern for procedure: an audiology assistant graded 73 of its own practice runs against 21 criteria, turned each miss into a "trigger, action, effect" rule, kept 19 rules, and won all 58 blind-rated cases against practicing clinicians. Adding reference facts through RAG did not help there; the model lacked procedure, not knowledge.

Variants that tested well

1. Rubric diagnosis loop. In PEEM you save the prompt and its response, score the pair on the nine axes, rewrite the prompt from the rationales and repeat. Two or three rounds raised accuracy by up to 11.7 percentage points across 7 benchmarks and 5 models, with no training.
2. Grow a playbook instead of rewriting it. Agentic Context Engineering (ACE) splits the job into a generator, a reflector and a curator, and adds each lesson as a small bullet "delta". The authors watched one full rewrite collapse an 18,282-token context to 122 tokens while accuracy fell from 66.7% to 57.1%. With deltas, ACE beat strong baselines by 10.6% on agent tasks and 8.6% on finance tasks.
3. A growing regression set. Data-model co-evolution turns every failure into a saved test case, generates 3-5 near-neighbours to see whether it is a one-off or a pattern, edits the prompt, then runs all accumulated cases through the old and new versions side by side.
4. Guidelines repaired from errors. In guideline repurposing for LLM annotators, a second model reads the annotator's mistakes against gold labels and rewrites the labeling guide; F1 rose from 0.36 to 0.58 in 3-5 iterations.
5. Tie-breaker rules for confused categories. Confusion-aware classification collects the category pairs the model keeps mixing up and writes one explicit distinguishing rule per pair into the prompt.

When it helps and when it hurts

Meta-prompting pays off on prompts you reuse: templates, classifiers, team instructions, agent system prompts. For a one-off question the PEEM authors note that two requests per iteration are not worth it.

The counter-evidence is about who controls the edits. In the Promptimizer user study, automatic ProTeGi-style optimization and a human-led rubric loop reached comparable accuracy, 63-64% on held-out comments, yet all 16 participants preferred the human-led loop: free automatic rewrites turned the prompt into a long, repetitive wall of text nobody could debug. ACE describes the same failure from the other side as brevity bias and context collapse. Free rewriting tends to drift either toward bloat or toward a generic summary; constrained edits, one rubric line or one rule per diagnosed error, keep the prompt legible and auditable.

Other limits the studies report: Spec Learning barely beat chance on a general-helpfulness dataset, so it needs a narrow task with articulable rules; ACE needs a feedback signal and degraded on tasks without ground truth; induced playbook rules can overfit to quirks of the test setup and need a manual pass. And the evaluator in every loop is itself a model, so the design rules on the LLM-as-a-judge page apply to the diagnosis step.

Evaluate the following prompt–response pair on 9 criteria.

**MY PROMPT:**
{the prompt you wrote}

**THE RESPONSE I GOT:**
{the LLM's response}

---

PROMPT EVALUATION (scale 1–5):

1. Clarity and structure — is the task clearly stated, 
   is the information in a logical order
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

2. Linguistic quality — grammar, precision 
   of wording, appropriate terminology
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

3. Fairness — neutral language, no 
   hidden assumptions or stereotypes
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

---

RESPONSE EVALUATION (scale 1–5):

4. Accuracy — factual correctness and logical 
   soundness
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

5. Coherence — logical structure, smooth transitions 
   between ideas
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

6. Relevance — how well the response solves the task 
   without digressions
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

7. Objectivity — neutral tone, no 
   bias or emotional clichés
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

8. Clarity of expression — easy to understand, no 
   ambiguity
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

9. Conciseness — gets the point across without fluff or repetition
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

---

Based on all the scores and rationales, rewrite my 
original prompt to fix every problem 
identified.

IMPROVED PROMPT:
[rewritten prompt]

You are a preference analyst. I'll give you {count} pairs of examples for the task: {task_description}.

Each pair: [task prompt] + [good version] + [bad version].

{example_pairs}

Find {5-8} principles that explain the difference between the good and bad versions.
For each principle:
- One sentence: what to do (or not do)
- The number of the pair that illustrates it best
If several pairs point to the same rule, merge them: that's an important signal.

I want to create a "playbook", a compact set of rules, for an AI assistant 
that will perform this task: {task}.

Let's do it as a 4-step loop:

STEP 1 — RUBRIC
Build a checklist of 8-12 quality criteria for this task. 
Each criterion is a specific behavior that can be graded.

STEP 2 — TEST RUN
Play the role of {assistant role} and perform the task in this scenario: 
{scenario}. Behave naturally and don't peek at the rubric.

STEP 3 — GRADE AND LESSON
Grade the result above against the rubric from step 1. Where the score is low, 
explain why and write a rule:
TRIGGER: [the situation where this applies]
ACTION: [what needs to be done]
EXPECTED EFFECT: [what it will achieve]

STEP 4 — ASSEMBLY
Add the rule to the "Task execution rules" list. 
If a similar rule already exists, don't duplicate it; refine the wording.

We'll repeat the loop (steps 2-4) several more times on different scenarios 
until we have 10-15 stable rules. I'll use the final list 
as a system instruction for real tasks.

Start with step 1. My task: {task}