Voting Ensembles: Making LLMs More Reliable by Voting Across Multiple Answers

INDEPENDENT REQUESTS:
Same question → Model (attempt 1) → Answer A
              → Model (attempt 2) → Answer A
              → Model (attempt 3) → Answer B
              ...
              → Model (attempt n) → Answer A

COUNT AND DECIDE:
Frequency of A: 35 out of 50
Frequency of B: 10 out of 50
Other: 5 out of 50

If k=40 (80% threshold) → Answer A did NOT reach the threshold → "No consensus"
If k=25 (50% threshold) → Answer A reached the threshold → Final answer: A

You can do all of this by hand in a chat, or ask the model to simulate multiple attempts.

Task: You're a lawyer checking whether a client can terminate a lease early without a penalty. Mistakes are expensive — bad advice will lead to a lawsuit.

Prompt:

Question: Can a tenant terminate a commercial lease early
without a penalty if the landlord has not fixed a roof leak within 2 months
of written notice?

Give me 10 independent legal opinions. Each opinion is a separate line of reasoning,
as if you were a different lawyer each time. Don't copy previous answers. After each opinion, state:
- Conclusion: YES / NO / UNCLEAR
- The legal basis (lease clauses, state landlord-tenant law, case law doctrines)

At the end, count:
- How many times the answer was "YES"
- How many times the answer was "NO"
- How many times the answer was "UNCLEAR"

If 8 out of 10 or more agree on one conclusion, that's the consensus answer.
If fewer, admit that the question is ambiguous.

Result:

The model returns 10 blocks of reasoning, each citing a specific legal basis. At the end comes the tally: for example, 9 out of 10 opinions agreed on "YES, the tenant can terminate under the constructive eviction doctrine and the landlord's duty to repair." That's high confidence — you can use it. If the split is 5-3-2 (YES-NO-UNCLEAR), there's no consensus — you need to talk to a live attorney or get more documents.

An LLM generates text probabilistically — a single question can have many possible continuations. If the question is deceptive (a wrong answer looks convincing) or bewildering (many more-or-less plausible options), the model will give different answers on different attempts.

At the same time, the model is good at generating many independent variants and spotting dominant patterns. Voting ensembles exploit this: if the right answer is obvious to the model, it shows up in most attempts. If the model isn't sure, the answers scatter, and the system honestly says "no consensus" instead of confidently hallucinating.

Control knobs:

1. Number of attempts (n): More attempts → a more accurate read on the spread. For simple questions 5-10 is enough; for critical ones, 20-50.
2. Agreement threshold (k/n):
   • Soft threshold (k=1, simple plurality) → maximum coverage, hallucination risk remains
   • Medium threshold (k=0.6n, 60% agreement) → a balance of coverage and trust
   • Strict threshold (k=0.8n, 80%+ agreement) → few answers, but almost no hallucinations
3. Model temperature: Higher temperature → more variability in the answers → easier to tell "confident" questions (consensus at any temperature) from "uncertain" ones (the spread grows).

Question: {your_question}

Give me {n} independent answers to this question. Each answer is a separate line of reasoning,
as if you were thinking about the question from scratch each time. Don't copy previous answers.

Format for each answer:
Attempt X: [your answer]

At the end, count how often each unique answer appears.

If {k} or more answers match, that's the consensus answer — give it as the final one.
If no answer got {k} votes, admit that the question is ambiguous and there's no consensus.

Final answer: [...]
Confidence: High / No consensus

Fill in:

• {your_question} — your question
• {n} — number of attempts (5-50, more for critical questions)
• {k} — agreement threshold (for high trust use 0.8n, for balance — 0.6n)

🚀 Quick start — paste into the chat:

Here's the Voting Ensembles technique for making answers more reliable.
Adapt the template to my task: [describe your task — e.g., "check whether a
medical diagnosis is correct", "choose between two business strategies", "solve a hard
math problem"].

Ask me questions to set up:
- How many attempts are needed (n)?
- What agreement threshold (k)?
- Do I need an explanation in each attempt or just the final answer?

[paste the template above]

The LLM will ask how critical the task is (to pick n and k) and what format works best for the answers — because different tasks call for different combinations: legal analysis needs a strict threshold (k=0.8n) and detailed explanations, while a math problem can get by with k=0.5n and short answers. The model will take the voting pattern from the template and adapt it to your task.

⚠️ Tokens: n attempts = n times more tokens. With n=50, a critical question will eat a big chunk of your budget. Use a high n only for questions where the cost of a mistake is high.

⚠️ Time: Multiple requests take time. If you need an instant answer, this method isn't for you.

⚠️ Doesn't fix fundamental ignorance: If the model doesn't know a fact (cutoff date, specific information outside its training), every attempt will hallucinate. Voting Ensembles help you detect uncertainty, but they don't add knowledge. For unknown facts, use web search.

⚠️ Systematic errors: If all the models were trained on the same data with the same error, the consensus will land on the wrong answer. The method helps against random hallucinations, not against systematic bias.

The team built a theoretical framework for questions with discrete answers. Each question is characterized by two parameters: δ (how deceptive the question is — is there an attractive wrong answer) and η (how bewildering the question is — how many random options there are). From this model they derived three theorems:

1. Accuracy is maximized at k=1 — the softest threshold always gives the most correct answers
2. Yield (coverage) is also maximized at k=1 — naturally, since with k=1 there's almost always a consensus
3. For large ensembles (n→∞), final accuracy tends to 100% if δ<0.5, to 0% if δ>0.5, and to 50% if δ=0.5

This means: if the right answer is even slightly more popular with the model than the wrong one, in the limit you get the right answer. If the wrong one is more popular, all the attempts converge on it.

The theory was tested on Llama3-70B-instruct and Llama3-8B-instruct:

• Synthetic arithmetic problems (multiplying multi-digit numbers, expressions with several operations)
• Real clinical data (500 echocardiograms — extracting ejection fraction, stenosis severity, regurgitation)

The results confirmed the theory: for arithmetic, raising k from 1 to 40 (out of n=50) lifted trust from 61% to 88% on multiplication, and from 48% to 65% on complex expressions. For clinical data — from 70% to 93% on mitral regurgitation. Meanwhile yield (coverage) predictably fell — from 90%+ to 60-70%, because a strict threshold filters out questions without a clear consensus.

The surprise: the method works even with chain-of-thought prompting. The worry was that CoT would add determinism and kill the variability, but no — CoT raised baseline accuracy (the model solves harder problems), and voting ensembles on top of CoT still boosted trust.

The practical takeaway: if you need maximum coverage (every question gets an answer), use k=1 (simple plurality). If you need maximum reliability and are willing to sacrifice coverage, raise k to 0.7-0.9 of n. The sweet spot is k≈0.6n: most questions get an answer, but the hallucinations get filtered out.

Increasing LLM Response Trustworthiness Using Voting Ensembles
Shir Goldfinger (University of Pennsylvania), Aparna Nair-Kanneganti (Harvard University), Trevor J. Chan (University of Pennsylvania), Emily Mackay, Brian Anthony (MIT), Alison Pouch (University of Pennsylvania)