TL;DR
Data-Model Co-Evolution is a methodology for systematically improving prompts, where every failure you find becomes a test case and every prompt revision is checked against all the accumulated examples. Instead of chaotic "eyeball tuning," you build a living test set that grows along with the prompt.
Usually the prompt and the tests live separately: tweak the prompt → check it on a couple of examples → seems to work → forget what you checked. A week later there's a new failure, you tweak the prompt again → something that used to work breaks. Regressions (fix one thing, break another) are a constant headache. The problem is that the examples don't accumulate, and the prompt changes blindly.
The method flips the process: found a failure → recorded it as a test → explained why it's wrong → generated similar examples to check generalization → updated the prompt → ran all accumulated tests. The prompt (model) and the examples (data) evolve together. Hence the name — co-evolution.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
A 5-step loop, each step doable in a regular chat:
STEP 1: Generate problem examples
→ The LLM creates inputs that may expose weaknesses in the current prompt
STEP 2: Labeling + rationale
→ You grade the result (OK/BAD) + write WHY it's wrong
STEP 3: Neighborhood probing
→ You generate 3-5 similar examples to check: a one-off failure or a pattern?
STEP 4: Prompt revision
→ Based on the rationales, you update the prompt (add a rule, clarify a boundary)
STEP 5: Regression testing
→ You run ALL accumulated examples against the new prompt version
→ Compare with the old version: what improved, what brokeThe key point: All examples from steps 1-3 are saved and used in step 5. The test set grows with every cycle.
Example in Practice
Task: You moderate comments on a Discord server about startups. You need to let harsh criticism of ideas through, but block personal attacks and unfounded accusations. The line is blurry — sometimes criticism of a business model sounds hurtful but is actually constructive.
STEP 1 — Generate problem cases:
Current prompt:
You are a moderator for a startup community. Allow constructive criticism of ideas.
Block personal insults and unverified accusations.Generation prompt:
Here is the moderator prompt:
[your prompt]
Come up with 5 user comments that are borderline —
it's unclear whether to allow or block them. The comments should be ones
where the rule doesn't apply in an obvious way.Result: The LLM produces comments like:
- "The founder clearly doesn't understand basic economics, this is a flop"
- "Yet another overconfident founder with zero experience barging into a crowded niche"
STEP 2 — Labeling:
Run each comment through the moderator. For example:
Here is the moderator prompt: [...]
Here is the comment: "The founder clearly doesn't understand basic economics, this is a flop"
Should the moderator allow or block it?The moderator blocks it. You think that's wrong — it's criticism of the idea, not the person.
Rationale (write it yourself or ask the LLM to suggest one):
Wrongly blocked. The comment criticizes the IDEA (the economic model),
not the founder as a person. The word "clearly" sounds harsh, but the substance is an assessment of the business logic.STEP 3 — Neighborhood probing:
Prompt:
Here is an example comment: "The founder clearly doesn't understand basic economics, this is a flop"
My rationale: [your rationale]
Generate 3 similar comments that:
- Keep the same trigger (criticism via an assessment of competence in the subject area)
- Vary the wording and context
- Do NOT widen the scope of the problem (only criticism of ideas via competence)Result:
- "Who even launches an MVP without unit economics? That's day-one stuff"
- "You can tell this person knows nothing about marketing, the product is doomed"
- "The financial model is a disaster, obvious to anyone who's ever calculated retention"
Run them through the moderator → if they all get blocked, it's a pattern. Label them BAD (wrong moderator decision).
STEP 4 — Prompt revision:
Based on the rationales, update the prompt:
You are a moderator for a startup community. Allow constructive criticism of ideas.
Block personal insults and unverified accusations.
IMPORTANT: Criticizing the founder's competence IN THE CONTEXT of a business decision is OK.
Examples that are OK:
- "The founder didn't account for unit economics"
- "The financial model is a disaster"
- "You can tell they know nothing about marketing"
Block only if:
- It's a personal attack WITH NO connection to the idea: "idiot", "fraud"
- It's an unfounded accusation: "definitely stealing", "obviously a scam"STEP 5 — Regression testing:
Take all the examples from steps 1-3 (say, 12 by now) + examples from previous cycles (another 8).
Prompt:
Here is the OLD version of the moderator prompt: [...]
Here is the NEW version of the moderator prompt: [...]
Here are 20 example comments with the correct decisions:
[list with OK/BAD labels]
Run each example through BOTH versions of the moderator.
Output a table: example | old version | new version | correct?
Calculate the accuracy for each version.Result: You see that the new version fixes 4 failures from step 3 but breaks 1 old case. Either you keep fixing the prompt, or you decide the trade-off is acceptable.
Why It Works
The LLM weakness: Models are bad at holding implicit policy boundaries. A prompt like "allow criticism, block insults" → the model interprets it its own way and often plays it too safe. You mean one thing, the model does another. Without examples, the model fills in the gaps on its own.
The LLM strength: Models learn very well from concrete examples with explicit rationales. Few-shot learning works. But give it 1-2 examples and the model will memorize them rather than generalize. You need coverage of edge cases.
How the method leverages this: Instead of a single example, you provide a family of similar cases (neighborhood probing). The model sees the pattern: "in all 4 cases, criticizing competence = OK." You add that rule to the prompt with examples → the model generalizes better.
Regression testing solves the biggest headache: "fixed one thing, broke another." You see explicitly how changes affect the entire set. No guessing — you measure.
Control Levers
1. Neighborhood size (how many similar examples to generate):
- 3-5 examples → fast, for simple rules
- 10-15 examples → deeper, for blurry boundaries
- The trade-off: speed vs confidence in generalization
2. The "similarity" criterion during generation: You can vary the instruction:
- "Keep exactly the same structure, change only the words" → a narrow test
- "Keep only the trigger, change the context" → a broad generalization test
3. Regression testing mode:
- Automatic (LLM-as-judge): fast but can be wrong → use it for a rough assessment
- Manual: slow but accurate → use it for critical cases
- Hybrid: the LLM proposes, you check the disputed ones
4. Loop stopping condition:
- A fixed number of iterations (5 cycles)
- Reaching a target accuracy (95% on the test set)
- Running out of request budget
- "I can't find any more failures" — a subjective criterion
Prompt Template
The method consists of 5 prompts, one for each step. Below are universal templates.
Step 1: Generate problem examples
Here is an instruction prompt for an AI:
{current_prompt}
Come up with {N} examples of user input that:
- Sit on the edge of the rules (it's not obvious how to handle them)
- May expose weaknesses or gaps in the instruction
- Vary in wording and context
Output only the list of examples, no explanations.Where to use it: {current_prompt} — your current instruction for the AI. {N} — usually 5-10.