Apply to my task
arXiv:2607.28439 ★ 83 Jul 30, 2026

ESPP: A Panel of Personas with a "Past" Instead of a Single LLM Judge

Key result

Five different personas in a prompt almost always deliver the same verdict: the LLM simply fits its justification to the 'you're a skeptic' label. The ESPP method lets you get genuinely different, well-grounded opinions from 'user types' in a single chat, with no focus groups and no code. The fix is simple: before scoring, each persona answers questions about its past experience; that's its memory, and the score must not contradict it. The trick: it's not the character description that keeps the model in check, but facts on the record — a label is easy to work around, your own words from earlier in the conversation are not.

01

TL;DR

ESPP (Evidence-Grounded, Social-Weighted Persona Panel) is a method for evaluating any content through several personas with a pinned-down history: they first rate the product individually, then exchange opinions under a strict rule, and finally their scores are combined into a weighted total. The point isn't that the LLM "plays roles" (every model can do that) but how those roles are anchored before the evaluation and how they're allowed to change their minds.

The study's main finding is surprising: if you simply ask an LLM to rate something on behalf of five different "user types," the result barely differs from a single opinion. The model just fits its reasoning to the persona label ("you're a skeptic" → any score is easy to justify with skepticism). Even running five independent passes and averaging gives a tiny gain. The real accuracy boost comes from one thing only: forcing the persona, before the evaluation, to answer concrete questions about its own experience, and then requiring the score not to contradict those answers.

ESPP handles this in three steps: (1) anchor each persona with specific answers to questions; that's its "memory," not just a personality tag; (2) let the personas see each other's scores and change their minds, but only if the argument is close to their own position AND touches a topic that personally matters to them; (3) build the final score not as a simple average, but weighted by each persona's expertise and relevance to the task.

Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.

02

Method Blueprint

STEP 0 (in the prompt): Create 5 personas with different traits (openness,
anxiety, risk appetite, etc.) + demographics + an experience profile.
For each persona, generate 3-5 questions about its experience with
similar products, plus its own answers → this is the persona's "memory."

STEP 1 (same or next prompt): Each persona independently rates
the product on several criteria, citing ITS OWN answers from Step 0
→ a score + justification from each persona.

STEP 2 (next prompt): The personas see all the other scores and justifications.
They change their opinion ONLY if:
(a) the other score is close to their own (not radically different)
(b) the argument concerns exactly what this persona cares about
→ revised scores.

STEP 3 (same prompt): The revised scores are combined with weights:
personas with more "expertise" in the topic and more "similarity" to
the target audience weigh a bit more → final score + spread of opinions.

All 4 steps can be run in one long chat as consecutive messages, with no code and no API.

03

Example in Practice

Task: A startup is launching a subscription service that delivers ready-made lunches to offices. Before running ads, it needs to check the landing page copy without relying on its own opinion or paying for a focus group.

Prompt:

Create 5 reader personas to evaluate the landing page copy below. For each,
specify: age, profession, attitude toward risk, how busy they are.

Then for each persona, come up with 3 questions about their past experience with
food subscriptions/delivery and answer them in the persona's voice. This will be
their "memory," which they must rely on when evaluating.

After that, each persona should rate the landing page copy (below) on
5 criteria: clarity, trust in the brand, ease of subscribing,
sense of control (can you cancel easily), transparency of terms.
Scale 1-5 with a short explanation that cites their "memory."

Then show each persona the others' scores and explanations. Let a
persona change their opinion ONLY if the other argument concerns a
topic that personally matters to them (judging by their character and answers),
and the neighbor's score isn't very different from their own. Otherwise,
they stick with their original view.

At the end, compile the overall assessment: calculate the average score for each
criterion, but note separately where the personas agreed and where
disagreements remained.

Landing page copy: {text}

Result: The model will produce profiles of 5 personas with their "history," then 5 independent scores with justifications, then a round showing who changed their mind and why (or didn't), and finally a summary table of scores showing which points had the widest disagreement (for example: everyone agrees the copy is clear, but they split on trusting a new brand).

04

Why It Works

An LLM that's simply told "be a skeptical user" has no real constraints. It can fit any justification to the "skeptic" label, because the label barely narrows the space of possible answers.

But LLMs are good at sticking to what has already been said in a conversation. If, before scoring, the persona has already "answered" concrete questions about its experience, those answers become an anchor. It's harder for the model to write a score that contradicts facts already on the record than one that contradicts an abstract personality label.

The method exploits this through anchoring with history (Steps 0-1) and through selective influence in Step 2: a persona changes its mind not because of any argument, but only one that falls within its own area of interest. That keeps the discussion from sliding either into unanimity (everyone converges to the average) or into chaos (nobody listens to anyone).

Control levers: - Number of personas (5 in the original) → drop to 3 for a quick check, raise to 7-8 for a finer picture of disagreements - The rule "change your mind only if the argument is on your topic and the score is close" → loosen or tighten it if you want more/less convergence of opinions - Weights in Step 3 (expertise, relevance) → state explicitly which persona is the "lead expert" for your task - Questions for the persona's "memory" → swap in questions about your specific product/niche

05

Prompt Template

I want to evaluate {object of evaluation} from the perspective of several types of people.
Evaluation criteria: {list of criteria, 3-5 items}.

Step 1. Create {number, e.g. 5} evaluator personas with different personality
traits, ages, professions and attitudes toward risk. For each one,
come up with 3 questions about their personal experience with {product category} and give
answers in the persona's voice. This will be their "memory" for the following steps.

Step 2. Have each persona rate {object of evaluation} on each criterion
(scale 1-5) with a short justification. The justification must cite
their "memory" from Step 1, not generic phrases.

Step 3. Show each persona the other personas' scores and justifications.
Let a persona revise their score ONLY if: the other argument
concerns a topic that matters specifically to them (judging by character and memory),
and the other score isn't very different from their own. Otherwise, they
keep their previous score.

Step 4. Compile the final score for each criterion. Personas with
more relevant experience should weigh slightly more when calculating
the average. Separately indicate where the personas agreed and where
serious disagreements remained.

{object of evaluation — text, product description, idea}

Placeholders: {object of evaluation} — what you're evaluating (copy, product, idea), {criteria} — which parameters to evaluate on, {number} — how many personas, {product category} — context for the questions about past experience.

🚀 Quick start — paste into the chat:

Here's a template for the ESPP method (a panel of evaluator personas). Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask which criteria matter for your task and which types of people to include in the panel, because that determines whether the personas will actually disagree (otherwise the method collapses into one averaged voice).

06

Limitations

⚠️ It's a long process: the full cycle (creating personas, memory, scoring, exchanging opinions, aggregation) requires a long prompt or several messages, so it's not suited to a quick one-line check.

⚠️ Narrow range of final scores: in the study, even very different models received similar overall scores. The method is good at revealing where exactly opinions diverge, but doesn't produce sharp contrasts in the overall score. Don't expect it to say "this is bad and this is great"; expect "everyone agrees here, and opinions split there."

⚠️ Quality depends on how well-designed the personas are: if the personas are shallow or their "memory" is inconsistent, the method degrades into ordinary role-play with no real benefit.

07

Resources

Zheng Wu, Yibo Luo, Pu Zhang, Cheng Yang, Zhuosheng Zhang (Shanghai Jiao Tong University, ByteDance Inc.), "Beyond a Single Judge: Simulating Social Persona Panels for Generative UI Evaluation". Code: https://github.com/Wuzheng02/ESPP

Research digest

Key result

Five different personas in a prompt almost always deliver the same verdict: the LLM simply fits its justification to the 'you're a skeptic' label. The ESPP method lets you get genuinely different, well-grounded opinions from 'user types' in a single chat, with no focus groups and no code. The fix is simple: before scoring, each persona answers questions about its past experience; that's its memory, and the score must not contradict it. The trick: it's not the character description that keeps the model in check, but facts on the record — a label is easy to work around, your own words from earlier in the conversation are not.

Principle

Three steps, not a single 'rate this as a skeptic' request. Step 1: the persona answers 3-5 questions about its experience; that's its memory. Step 2: the score must cite that memory, not generic phrases. Step 3: the personas see the others' scores, but change their minds only if the argument is close in position and falls within their personal interests. Without this rule, the discussion either slides into identical averaging or turns into a free-for-all where nobody listens to anybody.

Why it works

An LLM easily sidesteps an abstract label: the word 'skeptic' barely narrows the space of answers, you can fit anything under it. But its own words, said five turns earlier in the same conversation, are harder to dodge; the model holds on to what it has already written. Memory works as an anchor: the character is already pinned down by facts, rather than the score being fitted to the character. Without this step, five personas really do produce the same result as one — the researchers tested it.

When to apply

Testing copy, landing pages, offers, business ideas → when you need to quickly gauge how different audience types react without a real focus group, especially when what matters isn't the overall score but where exactly opinions diverge. Not suited to an instant one-line check: the process is multi-step and long.

Mini recipe

1. Create personas with memory: define 5 personas (character, age, profession), and for each, 3-5 questions about its past experience with a similar product plus answers in its voice. 2. Memory-backed scoring: each persona rates the object on the criteria, always citing its own answers, not generic phrases. 3. Selective exchange of opinions: show the personas the others' scores. Allow a change of mind only if the argument is on that persona's topic and the neighbor's score is close. 4. Weighted total: compile the final score weighted by each persona's experience, and note separately where everyone agreed and where disagreements remained.

Examples

[BAD]: `Rate this landing page copy on behalf of five different user types: skeptic, enthusiast, conservative, budget-minded, impulsive` [GOOD]: `Create 5 personas. For each, come up with 3 questions about its experience with food subscriptions and answer them in its voice; that's its memory. Then have each one rate the landing page, citing that memory rather than generic phrases. After that, show the personas each other's scores, and let them change their minds only if the argument concerns their own topic`

In plain words

Ordinary models judge content like an average of the entire internet, and the result is sterile and useless. The ESPP method changes the mechanics themselves: instead of one "smart" judge, you build a whole panel of different characters, each with their own backstory and quirks. The key is that the LLM doesn't just pretend to be someone; it gets locked onto specific experience that it can't ignore when delivering a verdict. That turns evaluation from reading tea leaves into a simulation of a real market.

It's like putting a hardened skeptic, an excitable tech geek and a budget-minded parent in one room to discuss your product. But there's a catch: if you just ask them to "be themselves," they'll start telling you what you want to hear. In ESPP each one gets a "passport" with past experience, and they have to speak up on their own first, then hear the others under a strict protocol, and only then give a final score. Democracy on steroids, where every opinion is weighted and backed by facts, not just the model's mood.

The whole trick rests on three pillars: evidence anchoring (a persona can't just say "I don't like it"; it has to cite its background), social weighting (accounting for whose opinion carries more weight in the group) and strict discussion rules. If you're testing a food-delivery landing page, the "office worker" will push for speed while the "health nut" pushes for calories. The method makes the model knock these positions together, surfacing hidden conflicts of interest that a plain "check my copy" prompt will never find.

Although the method was tested on interfaces, the principle is universal. You can point it at ad creatives, video scripts or even political platforms. Anywhere the outcome depends on how different groups of people perceive something, a persona panel will outperform a single expert. It lets you predict a product flop before you burn your budget on real tests. GEO and personalization demand exactly this approach: understanding not the "average user" but specific audience segments.

Bottom line: stop asking ChatGPT "is this good copy?" Build an ESPP panel, give the personas biographies and make them argue with each other. It's the only way to squeeze real criticism out of a model instead of an obligatory compliment. 15 minutes of simulation setup saves weeks of revisions after a failed launch. Either you test your product on virtual personas now, or on real customers later, with your own money.

AI lab assistant

Apply this study to your task

Describe what you're working on. AI takes the method and the measured results from this paper and adapts them to your case, or writes a ready prompt built on them.

Based on ESPP: A Panel of Personas with a "Past" Instead of a Single LLM Judge arXiv:2607.28439

Sign in to apply this study New accounts get 50 free credits.

Related techniques

Role and Persona Prompting