Self-consistency and answer ensembles: what the research shows

What self-consistency is

A single model response is one draw from a distribution of possible answers. Ask again and you may get a different one. Self-consistency turns that variability into information: generate several answers independently, then aggregate them.

The family has three moving parts:

• Where the answers come from: the same prompt resampled, the question reworded, different roles or reasoning styles, or different model families.
• How they are aggregated: majority vote, a judge that picks one, a reviewer that checks the disagreements, or a synthesizer that writes a new answer from the pieces.
• What happens without agreement: accept the plurality anyway, or admit the question is unresolved.

How much one draw varies is easy to underestimate. In a creativity study, 10-34% of idea quality came from sampling randomness alone; in data extraction, the same prompt on the same article swung from 95% to 78% accuracy between restarts.

Why agreement carries information

When a model knows something, it tends to produce it under any phrasing; when it is guessing, the guesses scatter. That is the core mechanism, and several papers measured it directly.

• The voting-ensembles study (Llama3-70B and Llama3-8B) raised the required agreement threshold and watched trust rise from 61% to 88% on multiplication and from 70% to 93% on echocardiogram extraction, while coverage fell because uncertain questions got no answer.
• Disagreement-based abstention asks the same factual question directly and in decomposed steps; for top models, accuracy was 50 times higher when the two answers agreed than when they disagreed.
• For agents, behavioral consistency across repeated runs predicted correctness: tasks with 1-2 distinct solution paths were solved at 80-92%, tasks with 6+ paths at 25-60%.

The flip side: if the model holds a systematic misconception, every sample repeats it and the vote confirms the error. The voting paper notes that the theory only converges on the right answer if the right answer is already more likely than the wrong one.

Decorrelate the samples

The strongest results came from making the answers fail differently, not from adding more of them.

• Different models. Agent-diversity scaling found two differently configured agents consistently beat 16 identical ones, because identical agents produce correlated outputs. In the LLM jury study, isolated models from different families voting on math tasks outperformed trained process reward models; the effect was weaker on graduate-level science, where models share textbook misconceptions.
• Reworded questions. SC-MoA rephrased each question five ways on one model at temperature 0, then synthesized the full reasoning chains. Synthesis fixed errors almost 2.5 times more often than it introduced them on GPQA-Diamond and 3.2 times on BBH-3.
• Different reasoning modes. CoT-PoT compares a verbal solution with executed code; 78.6% of problems were settled with exactly two requests.
• Different phrasings for ratings. Prompt perturbation averages scores from several wordings; one word change moved correlation with experts from 0.176 to 0.279. Ensemble scoring with eight independent ratings added +9.8 pp of judging accuracy.

Vote, pick or merge: where the studies disagree

Once you have several answers, the aggregation step matters as much as the samples, and the papers do not agree on the best one.

• Pick, don't mix. In the selection-bottleneck study (42 tasks across code, math, ethics, science, creative writing and more), synthesizing answers from a diverse team lost to a single model on 82% of tasks, while a judge that picked one answer beat it on 81%. The author's explanation: merging averages away the sharp ideas of the best answer.
• Mix, don't pick. Synthesize, don't select found that asking a model to pick the best legal or medical answer was close to a coin toss, while compiling one answer from all drafts was better, though it still captured only about 40% of the available quality. FusioN reported synthesis beating best-of-N by +2.5% win-rate on open-ended prompts, with N=2 fusion matching best-of-N at N=4, provided the fusor is a large model. Mixture-of-Agents measured +7.1 pp over a single answer and +2.7 pp over voting, with one synthesis round.
• A reading that fits both: selection wins when candidates differ sharply in quality and one is clearly best; synthesis wins on open-ended text where each draft has different strong parts. Neither paper tested the other's setup, so treat this as a working hypothesis.
• Constrain the judge. For tasks with a checkable answer, a free judge can be worse than the vote: in EL-DGR a full-authority judge scored 10 points below majority vote on a held-out set, and became best only when allowed to overrule the majority with an exact quote or recomputation. Divergent-convergent reasoning uses a strict reviewer told to check whether the minority answer is right; that recovered correct answers voting missed, and peer debate scored almost 40 points lower.

When agreement misleads

Consensus is evidence only when the answers are independent. SAVER shows three versions of an answer can share the same hidden assumption and recommends auditing reasoning against named violation types rather than trusting agreement. Synthetic consensus studied the human side: when several GPT-4o agents converged, participants' confidence rose even though no new evidence was added. Gradual convergence inflated confidence most.

Practical limits: N samples cost N times the tokens; sampling does not add knowledge the model lacks; and on subjective tasks a majority is an averaged opinion, not a correct answer. For checking a single answer without resampling, see verification prompts; for agents that argue before deciding, see multi-agent debate.

Question: {your_question}

Give me {n} independent answers to this question. Each answer is a separate line of reasoning,
as if you were thinking about the question from scratch each time. Don't copy previous answers.

Format for each answer:
Attempt X: [your answer]

At the end, count how often each unique answer appears.

If {k} or more answers match, that's the consensus answer - give it as the final one.
If no answer got {k} votes, admit that the question is ambiguous and there's no consensus.

Final answer: [...]
Confidence: High / No consensus

You are a judge. Your task: pick the best {content type}
out of the {number} provided. Do NOT merge the variants into one.
Do NOT rewrite. Just pick the winner.

Evaluation criteria:
- {criterion 1}
- {criterion 2}
- {criterion 3}

---
VARIANT 1:
{variant 1}

---
VARIANT 2:
{variant 2}

---
VARIANT 3:
{variant 3}

---

Assessment of each variant: 1-2 sentences against the criteria.
Winner: VARIANT [N].
Reason for the choice: one sentence.

You are a strict reviewer. Below are {N} independent solutions
to the same problem.

1. FIND CONSENSUS: steps that repeat across most solutions -
do a quick check on them.
2. FIND DISAGREEMENTS: points where the solutions diverge -
dig into them deeply.
3. SYNTHESIZE AN ANSWER: assemble a solution from the verified
steps + your analysis of the contested spots. Explicitly check
whether the MINORITY solution might be right - don't dismiss it
just because it's alone.

Solutions:
{insert all N solutions}

Give the final, well-reasoned answer.