Voice-Pool Audit: How to Catch an LLM Passing Off Quotes From Your Own Pitch as Criticism

STEP 1: Define the persona and context (who the person is, their experience in the category) → text block
STEP 2: Show the concept/pitch separately → text block
STEP 3: Set a fixed list of reason categories (e.g. price, trust, safety, habit) → checklist
STEP 4: Ask the model to play 3 voices (skeptic, neutral, enthusiast), and for each one:
        a) give a verdict (would buy / would not buy)
        b) explain it using the checklist categories
        c) explicitly NOT reuse the pitch's own wording as a reason
STEP 5: Compare the voices: if all three use the same words from the pitch as "reasons," that's a sign of a fake simulation

You can run all the steps in a single prompt, as long as the model can keep the roles cleanly separated inside one answer.

Task: A startup founder is preparing an investor pitch and wants to test in advance how skeptical venture partners will react to the arguments before walking into the real meeting.

Prompt:

Here is the startup's pitch:
"{pitch text}"

Play three investors reacting to this pitch. For each role:
1. Give a verdict: invest / pass / need more data
2. Explain the reason strictly through these categories (mark each as "plus," "minus," or "didn't matter"):
   - unit economics (trust in the numbers)
   - market (realism of its size and growth)
   - team (trust in execution)
   - timing (why now)
   - competitive advantage (is the moat real)
3. IMPORTANT: do not take a reason straight from the pitch text. If the pitch says "the market is growing 40% a year," don't write "the market is growing"; write whether the investor BELIEVES that number and why.

Roles:
- Skeptic: looks for where the pitch is weak and what hasn't been verified
- Neutral: weighs pros and cons equally
- Enthusiast: looks for what could convince them to get in

Don't mix the roles; output an answer for each one separately.

Result: The model returns three separate blocks, each with a verdict and a breakdown across the 5 categories. If in all three blocks the reasons match the pitch's own wording ("the market is growing 40%, so yes"), that's a sign the model isn't reasoning, just paraphrasing. If the skeptic finds real weak spots that aren't in the pitch text (say, "unclear how LTV was calculated"), that's a sign of substantive criticism.

LLMs are trained to be helpful and to go along with the logic of the input text. It's a kind of context-matching that shows up as hidden flattery. When the model plays a "customer" or an "investor," by default it drags arguments out of the very text you showed it, because that's the most readily available material for a "plausible" answer.

The model's strength is that it holds a structure really well once you spell it out. Ask it to "sort the reason into these 5 categories" and it will honestly sort, even when a category "didn't matter." That forces it beyond a simple retelling of the text.

The method turns this love of structure against the flattery: a fixed checklist of reasons keeps the model from sliding into paraphrase, and several contrasting voices (the skeptic kept apart from the enthusiast) physically separate "for" and "against," instead of one averaged answer that usually comes out neutral-to-positive.

Control knobs: - Number of voices → add more angles (budget shopper, brand loyalist, first-time buyer) to fit your task - Checklist categories → swap in ones relevant to your niche (for a B2B pitch: ROI, implementation risk, sales-cycle length) - Echo ban → tighten the wording ("don't use a single word from the original text as an argument") for a stricter test

Here is the text/concept to evaluate:
"{concept_text}"

Play three people reacting to it. Roles:
- Skeptic: looks for weak spots and doubts
- Neutral: weighs pros and cons equally  
- Enthusiast: looks for what could convince them

Persona context: {who this person is, their experience/habits in the topic}

For each role:
1. Verdict: {agrees / declines / needs more information}
2. Break down the reason strictly by category (mark "for," "against," or "not involved"):
   - {category_1}
   - {category_2}
   - {category_3}
   - {category_4}

IMPORTANT: do not take a reason straight from the concept text. If it says "X," don't write "likes X"; write whether the persona BELIEVES the claim X and why.

Output the three roles as separate blocks, without mixing them.

Fill in: {concept_text} is the pitch, ad, or product description; {who this person is} is the target audience; {category_1-4} are the concrete decision criteria in your niche (price, trust, risk, convenience, and so on).

🚀 Quick start: paste this into the chat:

Here's a template for testing my idea/pitch on a simulated audience. Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask who the target audience is and which decision criteria matter in your niche, because without that the reason checklist stays empty, and without the checklist the model slides right back into paraphrasing your text.

⚠️ Not about causality: the method checks whether simulated reasons agree with real ones, but it doesn't prove that a reason literally CAUSES the behavior. It's a test of honest reasoning, not rigorous causal analysis.

⚠️ LLM-simulated reasons are still weak out of the box: even with a checklist structure, the model tends to repeat the source text's wording. An explicit echo ban reduces the problem but doesn't eliminate it.

⚠️ You need a prepared category checklist: the method doesn't work out of thin air. You'll have to think through in advance which 5-7 reasons actually matter in your niche, or the breakdown will be shallow.

⚠️ Tested on a small sample: 94 people and 3 concepts is a proof of concept, not a broad generalization. For narrow B2C tasks (ads, landing pages) the pattern probably transfers; for complex B2B decisions it hasn't been tested.

The researchers recruited 94 women from India (ages 22-40), who rated three sunscreen concepts: a budget Ayurvedic one, a mid-range dermatological one, and a premium Korean one. Each wrote an open-ended "why" comment in addition to rating her purchase intent.

The team coded these comments into 7 fixed reasons (price, trust in the evidence, safety, naturalness, and others) with a plus/minus/zero sign, using a pipeline of two LLMs (a small model codes, a large one checks the disputed cases), without access to the purchase rating itself, so the answer couldn't leak into the labels.

Then came a simple test: a model predicts purchase intent from the person's context plus the reason. Adding the real human reasons sharply cut the prediction error, from 0.86 to 0.63 on the MAE scale. Next, an LLM was asked to generate those same reasons without seeing the ratings or the human comments, and the LLM version of the reasons turned out worse than even randomly shuffled real reasons. The model simply retold the concept's selling points instead of voicing real objections. Separately, they tested a "voice pool" (skeptic/neutral/enthusiast) and found that the right voice is often in the pool, but without a deliberate selection step the system averages them out and loses the signal.