PEEM: Diagnosing and Automatically Improving Prompts Through Structured Evaluation

Everything happens in one chat, no code or API needed.

STEP 1: Write a prompt, get a response from the LLM
         → a {prompt + response} pair

STEP 2: New request — evaluate the pair on 9 criteria
         → 9 scores (1–5) + a rationale for each

STEP 3: Based on the rationales — rewrite the prompt
         → an improved prompt

STEP 4: Repeat steps 1–3 (usually 2-3 iterations)
         → a final prompt with the problems fixed

Task: An e-commerce marketer wants to improve a prompt for generating Amazon product listings. The first results look dull and don't sell.

Prompt:

Evaluate the following prompt–response pair on 9 criteria.

**THE PROMPT I USED:**
Write a product description for Amazon: wireless earbuds 
TWS, 30-hour battery, ANC, price $49.99.

**THE RESPONSE I GOT:**
TWS wireless earbuds with active noise cancellation. 
Battery life — 30 hours. Price: $49.99. 
Great sound quality for everyday use.

---

Prompt evaluation criteria (scale 1–5, where 5 is best):

1. CLARITY AND STRUCTURE — is the task clearly stated, 
   is the information in a logical order
   Score: [1–5]
   Rationale: [specifically what's wrong]

2. LINGUISTIC QUALITY — grammar, fluency, 
   precision of wording
   Score: [1–5]
   Rationale: [specifically what's wrong]

3. FAIRNESS — neutrality, no 
   hidden assumptions
   Score: [1–5]
   Rationale: [specifically what's wrong]

---

Response evaluation criteria (scale 1–5):

4. ACCURACY — factual correctness and logical 
   soundness
   Score: [1–5]
   Rationale: [specifically what's wrong]

5. COHERENCE — logical structure, transitions between ideas
   Score: [1–5]
   Rationale: [specifically what's wrong]

6. RELEVANCE — how well the response solves the task 
   without digressions
   Score: [1–5]
   Rationale: [specifically what's wrong]

7. OBJECTIVITY — neutral tone, 
   no emotional clichés
   Score: [1–5]
   Rationale: [specifically what's wrong]

8. CLARITY OF EXPRESSION — easy to understand, 
   no ambiguity
   Score: [1–5]
   Rationale: [specifically what's wrong]

9. CONCISENESS — gets the point across without fluff or repetition
   Score: [1–5]
   Rationale: [specifically what's wrong]

---

Based on all the rationales: rewrite the original prompt 
to fix the problems identified.

IMPROVED PROMPT:
[new prompt]

Result:

The model returns a 9-row table — a score plus a concrete explanation. For example: "Structural clarity: 2/5 — the prompt doesn't specify the output format, target audience or tone. The model is forced to guess." Or: "Response relevance: 2/5 — the response lists specs but doesn't explain the benefits for the buyer."

At the end you get a rewritten prompt with the fixes. You run it, put it through the evaluation again, and after 2–3 iterations you have a prompt that consistently produces better listings.

The LLM weakness: When you ask "improve my prompt" with no criteria, the model guesses what you actually need. It sees the desired result but doesn't understand at what level the problem lies: the task wording, the structure, or missing context. The edits end up being random.

The LLM strength: Models are great at evaluating against predefined criteria with an explanation. When the criteria are fixed in the prompt, the evaluation becomes reproducible and specific. That's what G-EVAL does well: a structured rubric yields a structured breakdown.

How the method uses this: PEEM gives the model a precise list of axes — not "rate the quality" (too abstract) but "rate structural clarity" and "rate conciseness" separately. Each axis is isolated, so the model doesn't blend criteria. The rationales become concrete editing instructions: not "the prompt is bad" but "the prompt doesn't specify the output format or audience."

Control knobs:

• Number of iterations → one round is enough for simple prompts; three or four for complex ones
• Set of criteria → you can drop "fairness" for technical tasks where bias doesn't matter, or add your own axis for the specifics of your task
• Rationale detail → add "give a concrete example of how to fix it" and you'll get ready-made edits, not just a diagnosis
• Splitting roles → you can have one chat only evaluate and another only rewrite. This reduces task blending

Evaluate the following prompt–response pair on 9 criteria.

**MY PROMPT:**
{the prompt you wrote}

**THE RESPONSE I GOT:**
{the LLM's response}

---

PROMPT EVALUATION (scale 1–5):

1. Clarity and structure — is the task clearly stated, 
   is the information in a logical order
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

2. Linguistic quality — grammar, precision 
   of wording, appropriate terminology
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

3. Fairness — neutral language, no 
   hidden assumptions or stereotypes
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

---

RESPONSE EVALUATION (scale 1–5):

4. Accuracy — factual correctness and logical 
   soundness
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

5. Coherence — logical structure, smooth transitions 
   between ideas
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

6. Relevance — how well the response solves the task 
   without digressions
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

7. Objectivity — neutral tone, no 
   bias or emotional clichés
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

8. Clarity of expression — easy to understand, no 
   ambiguity
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

9. Conciseness — gets the point across without fluff or repetition
   Score: [1–5]
   Rationale: [what exactly is wrong — be specific]

---

Based on all the scores and rationales, rewrite my 
original prompt to fix every problem 
identified.

IMPROVED PROMPT:
[rewritten prompt]

Placeholders: - {the prompt you wrote} — the original prompt, word for word - {the LLM's response} — the response that prompt produced

🚀 Quick start — paste into a chat:

Here's a PEEM template for evaluating and improving prompts. 
Adapt it to my task: {your task}. 
Ask me for the prompt I want to improve 
and the response it gave.

[paste the template above]

The LLM will ask for your original prompt and the response you got — because without the prompt+response pair there's no way to judge how well the prompt did its job.

⚠️ Subjective tasks: The "objectivity" and "fairness" criteria work poorly for creative tasks — there's no neutral reference point. For "write a funny post," objectivity isn't a criterion.

⚠️ Overhead: Each iteration is two requests (get the response + evaluate it). For simple tasks that's overkill. PEEM pays off where a prompt gets reused: templates, workflows, recurring tasks.

⚠️ The evaluator is an LLM too: The model can get the evaluation wrong. Especially when checking factual accuracy (the "Accuracy" axis) — if the LLM doesn't know the right answer, it won't catch the error. Fact-checking needs an external source.

⚠️ Different models score differently: There is correlation between evaluators (ρ = 0.68–0.85), but it's not perfect. Use the same model for evaluation within one project — otherwise the scores aren't comparable.

The researchers asked: can manual prompt review be replaced by a structured rubric with an LLM evaluator? To find out, they tested PEEM on 7 benchmarks (including math, question answering and reasoning) with 5 different models (Gemma, LLaMA, Qwen, GPT-4o-mini, Gemini).

First they checked whether PEEM scores match traditional accuracy. They got Spearman ρ ≈ 0.97 — a near-perfect match in how the models were ranked. That matters: PEEM doesn't invent a new scale; it correlates well with what's already being measured.

Then they checked robustness. Meaning-preserving paraphrases of a prompt (same idea, different words) produced stable scores — 76–80% of cases with no change in score. But deliberate semantic distortions (changing the meaning) immediately pushed scores down. This shows the metric responds to content, not to the surface look of the text.

The most interesting part is the rewriting loop. The researchers took weak prompts, evaluated them with PEEM, fed the rationales back to the model as rewriting instructions — and repeated this several times. The result: an accuracy gain of up to +11.7 percentage points, with no training and no labeled data. That beat a number of methods based on supervised fine-tuning and RL-based prompt optimization. The takeaway: interpretable feedback works better than blind optimization against a metric.

Query: How many oranges does John have if he starts with 8 oranges, 
eats 3, and then buys 10 more?

Response: 10 plus 5 is 15.

Response Evaluation:

Criterion | Score & Language Rationale

Accuracy    3: The final result (15) is correct, but the intermediate 
              calculation ("10 plus 5 is 15") is disconnected from 
              the original problem.

Coherence   2: The reasoning is disjointed and fails to logically 
              explain how the result was derived.

Relevance   3: The response partially addresses the question, 
              providing the correct final number but neglecting 
              the problem's context.

Objectivity 5: The response is neutral and factual.

Context: An example from evaluating a response to the oranges problem. It shows how PEEM catches a hidden problem: the answer is technically correct (15 = right), but the explanation is broken — "10 plus 5" doesn't follow from the problem. A traditional accuracy metric would give it 1/1. PEEM catches the gap.

💡 Adaptation: a single round without a sample response

If you want to evaluate a prompt before running it, ask the model to predict which evaluation problems will come up:

Evaluate this prompt on 3 criteria without running it:

MY PROMPT: {prompt}

1. Clarity and structure — is it clear what needs to be done?
   Score: [1–5] | Problem: [specific]

2. Linguistic quality — is the wording precise?
   Score: [1–5] | Problem: [specific]

3. Fairness — are there hidden assumptions?
   Score: [1–5] | Problem: [specific]

Rewrite the prompt, fixing all the problems:
IMPROVED PROMPT: [new prompt]

It's faster — no need to wait for a response before you start improving.

🔧 Technique: add your own evaluation axis → custom diagnostics

The standard 9 axes are universal. For specialized tasks, add your own:

10. FIT WITH THE BRAND'S TONE OF VOICE — 
    how much the response sounds like [brand name]:
    bold, witty, no corporate-speak
    Score: [1–5]
    Rationale: [specific]

This is especially useful for: content marketing with a signature voice, legal texts (a "regulatory compliance" axis), HR communications (an "empathy toward the candidate" axis).

🔧 Technique: ask only about the weak axes → save tokens

If you already know where the problem is (say, the responses are always off-topic), evaluate just one or two axes:

Evaluate only the RELEVANCE and CONCISENESS of the response.
Don't evaluate the other criteria.

[the rest of the structure stays the same]

Half the tokens, three times faster iteration.

PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses

Authors: Minki Hong, Eunsoo Lee, Sohyun Park, Jihie Kim
Affiliation: Department of Computer Science and Artificial Intelligence, Dongguk University, South Korea
Contact: [email protected]

Related methods: G-EVAL, GPTScore, ChatGPT-as-a-Judge, Chain-of-Thought prompting