Reflective Prompt Optimization (GEPA): auto-improving an agent's prompt by analyzing errors criterion by criterion

STEP 1: Run the agent on a small sample (20–30 tasks) → work records (traces)
STEP 2: Score each answer BY CRITERION separately → a written breakdown for every criterion
STEP 3: Reflector reads prompt + traces + breakdown → diagnosis + a new version of ONE component
STEP 4: Test the candidate on a separate validation sample → keep the best
STEP 5: Repeat; final step: test on DIFFERENT topics, multiple runs

In the study, steps 1–4 are automated in code. You can reproduce the logic by hand across a few chats (see the template).

Task: You're building an AI agent for a B2B sales team. It prepares briefs on client companies before calls: revenue, leadership, news, lawsuits. Every number needs a link to a source. Sometimes the agent gives links that are "on topic, but the numbers there are different," and sometimes it outputs citation numbers that don't match the source list. You want to improve its system prompt based on data, not gut feel.

Prompt (for a chat with a strong model in the "reflector" role):

You are a reflector: you improve the system prompt of an agent that writes company briefs with citations.

<CurrentPrompt>
{the agent's current system prompt}
</CurrentPrompt>

<Traces>
Task 1: Brief on Acme Logistics LLC (EIN ...)
Agent work record: {searches, pages found, final brief}
Criterion breakdown:
- link_opens: 4 of 5 (two links point to nonexistent pages)
- source_on_topic: 5 of 5
- source_supports_fact: 2 of 5 (revenue stated as 1.2 billion, the source page says 840 million)
- numbering: 5 of 5

Task 2: ...
</Traces>

Rules:
1. Find the CRITERION with the lowest score and use the records to explain which agent behavior drags it down.
2. Propose a GENERAL behavior rule. Do not copy specific facts, company names, or numbers from the tasks into the prompt.
3. Change only one block of the prompt: "search strategy" OR "report formatting."
4. Don't grow the prompt by more than 100 words. If you can cut something unnecessary, cut it.
Output: diagnosis → changed block → what you expect to improve.

Result: The model will give a diagnosis for the weakest criterion ("fact support"), citing specific traces. Then it will propose one general rule, for example about checking a number against the source text before putting it in the brief. Then it will rewrite one block of the prompt. You run the new version on separate tasks and compare it with the old one. You should also test it on companies from other industries, not the ones in the examples.

The LLM optimizer's weakness: if it only sees one final number, it doesn't know what exactly is broken. It has two options: guess, or overfit the prompt to the specific examples (by writing the answers into it). In the second case you get a gain on training tasks but none on new topics. That's exactly what the scalar score produced in the study: transfer went negative.

The LLM's strength: the model is good at reading a long work record and writing a clear natural-language diagnosis from it. It just needs to "see" which criterion is slipping.

How the method exploits this: the score is split into separate "levers" (link opens, source on topic, source supports the fact), and every low score comes with a reason. The reflector gets a concrete target and derives a general rule. Selection on a separate sample and across the candidate frontier weeds out bloated and overfitted versions.

Control knobs: - Number of criteria → more criteria give a more precise diagnosis but require more checks. The criteria must be independent. - "Don't copy facts from the tasks" → keeps memorized answer keys out of the prompt and saves words. - Length-growth cap → protects against bloat; in the study, longer versions consistently lost. - One component per step → makes it easier to see which edit helped. - Different topics in testing → shows whether the gain transfers or it's just overfitting.

You are a reflector. You improve an agent's system prompt based on records of its work.

<Role>Prompt diagnostician: you look for general causes of errors and don't overfit to individual examples.</Role>

<AgentTask>{what the agent does}</AgentTask>

<CurrentPrompt>
{block A: name}: {text}
{block B: name}: {text}
</CurrentPrompt>

<Criteria>
{criterion_1}: {what we check}
{criterion_2}: {what we check}
{criterion_3}: {what we check}
</Criteria>

<Traces>
Task {number}: {wording}
Work record: {what the agent did and what it output}
Criterion breakdown:
- {criterion_1}: {score}: {reason}
- {criterion_2}: {score}: {reason}
- {criterion_3}: {score}: {reason}
</Traces>

<ReasoningLogic>
1. Find the criterion with the lowest score.
2. Use the records to determine which agent behavior drags it down.
3. Formulate a GENERAL rule. Don't insert specific facts, names, or numbers from the tasks.
4. Pick ONE block of the prompt and rewrite only that block.
5. Check the length: growth no more than {word_limit} words; cut anything unnecessary.
</ReasoningLogic>

<Output>Diagnosis → the full changed block → which criterion should go up.</Output>

What to fill in: - {criterion_N}: 3–5 independent quality criteria for your task that you can score. - Traces: 3–6 tasks with the agent's real output and a per-criterion breakdown. You can do the breakdown by hand or with a separate judge chat. - {word_limit}: 50–150.

🚀 Quick start: paste this into a chat:

Here's a template for reflective prompt optimization. Adapt it to my task: {describe what your agent or prompt does}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what the agent does, which quality criteria matter, and which errors you've already seen. The criteria are essential, because the whole method rests on broken-down feedback. From them, the model will build both a scoring template and a meta-prompt for the revisions.

⚠️ One number instead of criteria: if you collapse the evaluation into a single score, the method does harm. Transfer to new topics goes negative, even while the gain on training tasks may look great.

⚠️ Memorized answer key: the reflector tends to copy specific facts from the examples into the prompt. This adds nothing and bloats the prompt. You need a ban in the meta-prompt and a human read-through of the prompt.

⚠️ Bloat: versions where the prompt kept growing got worse. Without a length cap, the method ruins the prompt.

⚠️ Noisy evaluation: on a small sample of 24 tasks, quality fluctuates. The gap between "on validation" and "on new topics" jumps around a lot from run to run. Don't trust a single run; you need several plus a test on other topics.

⚠️ Narrow evaluation: one task (citations in a research agent), one model family, and the judges are models too. The per-criterion ablations were done on two runs and show a direction, not precise numbers.

⚠️ Uneven gains: the win was real but not across the board: some tasks got worse. Runs varied a lot in effect size.

⚠️ The engineering side: in the study, the GEPA loop itself was run in code over hundreds of runs. In chats you can reproduce the principle by hand, but not the scale.

The team took a two-module deep-research agent: an orchestrator plans and hands out work, and a researcher searches a frozen corpus and writes a report with citations. Both system prompts were optimized. The reward was composite: a correct answer acts as a gate (a wrong answer zeroes everything out), followed by checks on every citation: does it open, is it on topic, does it support the fact, is it numbered correctly.

The key move was splitting the data by topic. 24 tasks for training and 24 for validation, 12 topics each, plus 500 tasks on 254 other topics for the final test. No topic appears twice. So no amount of answer memorization can help on the test. The test score rose from 0.702 to 0.781 (confidence interval [0.051; 0.107]) and replicated across three runs with an average gain of 0.107.

Next they tested a suspicion. In the best prompt, almost half the words were an "answer key" with facts from the training questions. The researchers tested it on the same questions and on new questions about the same articles. The answer key didn't even raise the answer-correctness score. In other words, the big gap between validation and test is noise and selection on a small sample, not memorization. On new questions, topics whose facts had "leaked" into the prompt gained less than the rest. The leaked facts are just dead weight.

Then they varied the type of feedback while keeping everything else the same: a coarse scale, triple scoring with averaging, and a single number instead of a per-criterion breakdown. Only the last option broke transfer: instead of +0.075 it came out at −0.021. More frequent or more precise scoring added nothing. Evaluation hacks (gains on validation without gains on test) coincided with the weakest judges. The authors also note that a hand-written version of the discovered recipe mostly reproduces the gain.

💡 Context adaptation: manually checking your own system prompt without GEPA. If you can't run the loop, take 5–10 real answers from your custom GPT and score them on 3–4 criteria with reasons. Then hand that to a chat using the template above. Even one manual iteration gives you a general rule for the prompt.

🔧 Technique: a leak checklist → a clean prompt. After any automated prompt revision, add this step:

Reread the new prompt. List every place that mentions specific companies, numbers, dates, or names from the examples. Delete them or replace them with a general rule. Report how many words were removed.

This closes off the "memorized answer key" risk from the paper.

🔧 Technique: test on other topics. Before accepting a new prompt, run it not on the tasks you showed the reflector, but on held-out tasks from a different area. If the gain only shows up on "familiar" tasks, it's overfitting.

• Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfitting by Elias Lumer, Sahil Sen, Vamse Kumar Subbiah, Kevin Paul. The authors publish the candidate prompts, diffs between versions, tables, and analysis.
• GEPA (Agrawal et al., 2025): the optimizer used in the paper; open-source implementation GEPA-AI.
• DSPy and MIPROv2: related approaches to automatic prompt optimization.
• Decagon (2026): recommendations on a strong reflector, a sample of 20–100 examples, and prompt bloat.
• Onweller et al. (2026): work on "present but not supporting" citations in research agents.