Verbalized Sampling: Distributions Instead of Single Answers to Beat Sameness

VS-Standard (one request):

Prompt: "Generate {k} answers to {task} with their probabilities"
→ The model returns a list: (option_1, probability_1), ..., (option_k, probability_k)

VS-CoT (one request):

Prompt: "Think step by step, then generate {k} answers to {task} with probabilities"
→ The model shows its reasoning, then a list with probabilities

VS-Multi (several requests):

Round 1: "Generate {k} answers to {task} with probabilities"
Round 2: "Generate {k} more answers with probabilities"
→ You collect N answers over ⌈N/k⌉ rounds

If you need N=30 answers with k=5, you make 6 requests of 5 options each.

Task: You're a copywriter at a marketing agency. You need creative slogan options for a new grocery delivery service, "FreshCart Express". The client wants to see variety, from a serious to a playful tone.

Prompt:

Generate 5 advertising slogan options for a grocery 
delivery service called "FreshCart Express", with a probability for each option. 
Show a variety of styles, from a businesslike to a playful tone.

Result: The model will return 5 slogans, each with a probability:

• A businesslike option (probability ~25%)
• A playful option (probability ~20%)
• An emotional, family-themed option (probability ~20%)
• An option focused on speed (probability ~20%)
• An unconventional creative option (probability ~15%)

Instead of getting one "safe" option like "Delicious, delivered fast", you see a spectrum of ideas, from conservative to bold. You can pick the best one or ask the model to develop the direction you like.

After alignment, LLMs suffer from excessive caution: when many answers are correct (creative tasks, open-ended questions), the model picks the safest and most typical one. This happens because the people annotating preference data favor the familiar, a result of cognitive effects like the mere-exposure effect (we like what we see often) and processing fluency (what's easy to process seems better).

An LLM's strength is its ability to model distributions. Trained on trillions of words, the model absorbed not only typical answers but also variability: what kinds of jokes exist, what stories people tell, how they phrase their thoughts. That variability is "hidden" after alignment, but not lost.

VS uses this strength to get around the weakness: when you ask for a distribution with probabilities, the model switches from "find the one best answer" mode to "show the range of likely answers" mode. The mode (the most likely output) changes: for the prompt "one joke" the mode is the most stereotypical joke; for the prompt "5 jokes with probabilities" the mode is a distribution of jokes close to what the model saw in training.

Control levers:

• Number of options k: higher k → more diversity, but quality drops at k>10. The sweet spot is k=5-7.
• Probability threshold: you can add "with probabilities below 40%" → even more diversity.
• CoT instruction: "think step by step" before generating → better quality at the same diversity.
• Temperature: VS is orthogonal to temperature; you can combine them (VS + temperature 1.2) for maximum diversity.

VS-Standard:

Generate {number} answers to the task: "{your_task}".
For each option, give its probability (as a percentage). 
(our addition, not tested).

VS-CoT:

Think step by step about possible approaches to the task: "{your_task}".
Then generate {number} answers with probabilities (as percentages).
(our addition, not tested).

VS-Multi (for each round):

Generate {number} answers to the task: "{your_task}".
For each option, give your confidence (as a percentage).

[After the first round, add:]
Generate {number} more options, different from the previous ones, with confidence.

What to plug in:

• {number}: usually 5-7 (the sweet spot for quality vs. diversity)
• {your_task}: a specific task ("write a joke about programmers", "suggest names for an EdTech startup")

🚀 Quick start: paste this into the chat:

Here's the Verbalized Sampling template. Adapt it to my task: [your task].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about the specific task, the number of options, whether you want step-by-step analysis (CoT), and how much diversity you need. It will take the pattern from the template and adapt it to your task, so you don't have to work out the details by hand.

⚠️ Tasks with a single right answer: On factual questions ("Capital of France?") or math with one solution, VS gives no advantage: mode collapse does no harm there.

⚠️ Quality at very large k: If you ask for 15-20+ options at once, the quality of the last ones drops. The sweet spot is k=5-7; for more, use VS-Multi (several rounds).

⚠️ Readability in dialogues: In the dialogue-simulation task, VS generates more complex text (a higher Flesch-Kincaid level) than real people do. This shows the method isn't yet perfectly calibrated for tasks where natural language matters.

⚠️ Requires manual selection: You get several options but have to pick the best one to use. That's an advantage (you see alternatives), but also an extra step.

The researchers took on mode collapse and asked: where does it come from? Instead of blaming the algorithms (RLHF, DPO), they dug deeper, into the preference data itself. The hypothesis was simple: human annotators are biased toward typical text. It's not malice but a cognitive quirk: the brain likes what's familiar (the mere-exposure effect), easy to process (processing fluency), and in line with expectations (schema congruity).

To test it, they took the HELPSTEER dataset, which rates answers on correctness (true usefulness) and helpfulness (the final reward). They formed 6,874 pairs of answers to the same prompt with equal correctness but different helpfulness. If annotators were neutral, helpfulness would depend only on correctness. But no: the model showed that the higher an answer's log-likelihood under the base model (i.e., the more typical the text), the higher its helpfulness, regardless of correctness. The typicality coefficient was α=0.57±0.07 (p<10⁻¹⁴). That's the proof: annotators systematically prefer the typical, even when it isn't substantively better.

Next they showed mathematically that if typicality enters the reward (r = r_true + α·log π_ref), then the optimal policy after RLHF sharpens the base model's distribution (π* ∝ π_ref^γ, where γ=1+α/β>1). When many answers are equally good (flat r_true), typicality becomes the tie-breaker → mode collapse.

They tested VS on 5 kinds of tasks: creative writing (poems, stories, jokes), dialogue (simulating donation conversations), open-ended questions (name US states), synthetic data generation (math problems), and random number generation. The surprise: VS works better on more powerful models: GPT-4.1 gains more from VS than GPT-4.1-mini. That runs against the intuition that a simple prompt should work the same everywhere. Why? Larger models understand the "distribution with probabilities" instruction better, don't suffer from cognitive burden (overload from a complex prompt), and have richer internal representations of distributions.

The main practical insight: VS isn't just a hack; it's a way to unlock what the model already knows. Alignment doesn't kill diversity, it "hides" it. VS gives you access back.

The paper's VS-Standard prompt for creative-writing tasks (poems, jokes, stories), quoted from the appendix; the joke topic goes in as the input prompt. Short form from the paper: "Generate 5 jokes about coffee and their corresponding probabilities."

Generate {num_samplings} responses to the input prompt. Each response should be 
approximately {target words} words.
Return the responses in JSON format with the key: "responses" (list of dicts).
Each dictionary must include:
• text: the response string only (no explanation or extra text).
• probability: the estimated probability from 0.0 to 1.0 of this response given 
  the input prompt (relative to the full distribution).
Give ONLY the JSON object, with no explanations or extra text.

Context: The researchers tested on 100 topical prompts from Reddit r/DadJokes ("Write a joke about [topic]"). For each prompt they generated k=5 candidates, collected N=30 total samples over several calls, and measured semantic diversity (1 - mean cosine similarity), lexical diversity (ROUGE-L), and quality with Claude-3.7-Sonnet as the judge.

💡 Adaptation for brainstorming business ideas: VS is great for idea generation where you want to see the full range of possibilities. Instead of "Suggest an EdTech startup idea", use:

Generate 7 EdTech startup ideas for the US market with probabilities.
Show variety: from conservative to bold, from B2C to B2B.
For each idea, give the probability that you would generate it (from 0.0 to 1.0, relative to the full distribution).

You'll get: traditional courses (~25%), an AI tutor (~20%), gamification (~15%), corporate training (~15%), EdTech for kids with special needs (~10%), niche certifications (~10%), a totally wild idea (~5%). Note what the numbers mean: in VS a probability is how likely the model is to produce that idea (how typical it is), not a forecast of its success or risk. The paper never tested VS as a success estimate. (The ideas and percentages above are our illustration.)

🔧 Technique: Probability threshold → creativity control

Add this to the prompt:

Generate 5 options with probabilities BELOW {threshold}%

Effect:

• Threshold 40% → you filter out the "safe" options and get more creative ones
• Threshold 15% → only the most unusual ideas
• Threshold 50% → a balance of typical and creative

Example:

Suggest 5 names for a coffee shop in Brooklyn with probabilities below 30%.
Avoid obvious options.

Instead of "The Coffee House" or "Morning Brew", you'll get bolder ones: "Brewology", "Jolt Theory", "Black & White", "Third Wave", "Bean Lab".

💡 Extrapolation: VS + temperature + negative prompting

Combine VS with a high temperature (1.2-1.4) and negative instructions:

[temperature 1.3]

Generate 5 stories that begin with "Without saying goodbye," with probabilities.

DO NOT use: romantic breakups, partners who suddenly vanish, 
typical dramatic plots.

Experiment with genres: sci-fi, absurdism, minimalism, social drama.

This pushes the model even further from stereotypes. VS provides the variety, temperature amps up the randomness, and negative prompting blocks the clichés.

Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity

• Website
• Blog
• Code on GitHub

Authors: Jiayi Zhang, Simon Yu, Derek Chong (Northeastern University), Anthony Sicilia (West Virginia University), Michael R. Tomz, Christopher D. Manning (Stanford University), Weiyan Shi (Northeastern University)

Related work cited in the paper:

• Padmakumar & He (2024), West & Potts (2025): the first studies of mode collapse after alignment
• Kirk et al. (2024): the PRISM dataset for pluralistic alignment
• Lu et al. (2025b): a Creative Writing benchmark for evaluating LLM creativity